A quantitative defense of linguistic methodology by Jon Sprouse & Diogo Almeidaa

It is often argued that the current divide between linguistic theory and other domains of language research can be traced to the unreliability of linguistic methodology, and the resulting unreliability of linguistic theory. Linguists rely upon informal acceptability judgment experiments as the primary source of linguistic data. These experiments tend to employ small samples, often composed of non-naïve participants, and generally forego the use of distracter items and inferential statistics in the analysis of the results. We present resampling analyses of a large acceptability judgment dataset that provides new quantitative evidence that traditional linguistic methodology is extremely reliable with very small samples, usually at the level of the individual participant. We also review the evidence that critics of informal linguistic experiments provide to justify their concerns, which shows that the evidence provided by critics turns out to mostly corroborate the reliability of informal judgments, and not undermine it. We conclude that there is no empirical, logical, or statistical reason to think that the informal experiments routinely performed by linguists are unreliable. In fact, we show evidence that these experiments might be, in some circumstances, much more powerful than formal experiments with naïve participants. Given the lack of evidence of problems with traditional linguistic methodology, we hypothesize that one potential reason for the recurrence of this debate is that the phenomena critics are particularly interested in often elicit effects that are very small and hard to detect in formal acceptability judgment tasks. This suggests that critics may be mistaking a property of specific phenomena for a property of the methodology. Taken as a whole, these results suggest that broad criticisms of linguistic theory based on the reliability of linguistic data are unfounded, and that methodological concerns should not influence the relationship between linguistic theory and other domains of language research.

Looks like a good new resource, if they keep it going. Two useful posts so far.

Baseline level of excitement because OMG it’s the world cup (on a scale of 1-10): 10.

Added excitement from what’s actually happening in the games: not quite enough.

Sources of excitement: every time Messi touches the ball, every third time Snijder touches the ball, every time a German player touches the ball, one shot by Ronaldo, Maicon scoring from an impossible angle, Chile’s mad attacking machine (especially #7 Sanchez), and one truly gripping match between Spain and Switzerland.

Robben, Drogba, Rooney: please get back in form.

Best coaching move: the wholesale importation of the German U21 team into the senior side.

Favorites now: Germany, Argentina, Netherlands, Brazil, probably Spain still, and — dare I say it? — Chile.

Predictive value of the first set of games: zilch (see Holland at Euro 08).

NASSLLI 2010 is Open for Registration!

Fourth North American Summer School in Logic, Language, and Information NASSLLI 2010

June 20-26, 2010

http://www.indiana.edu/~nasslli/

The North American Summer School in Logic, Language, and Information (NASSLLI) is a summer school with classes in the interface between computer science, linguistics, and logic.

After previous editions at Stanford University, Indiana University, and UCLA, NASSLLI will return to Bloomington, Indiana, June 20–26, 2010. The summer school, loosely modeled on the long-running ESSLLI series in Europe, will consist of a number of courses and workshops, selected on the basis of the proposals. Courses and workshops meet for 90 or 120 minutes on each of five days, June 21–25, and there will be tutorials on June 20 and a day-long workshop on June 26. The instructors are prominent researchers who volunteer their time and energy to present basic work in their disciplines. Many are coming from Europe just to teach at NASSLLI.

NASSLLI courses are aimed at graduate students and advanced undergraduates in wide variety of fields. The instructors know that people will be attending from a wide range of disciplines, and they all are pleased to be associated with an interdisciplinary school. The courses will also appeal to post-docs and researchers in all of the relevant fields.

We hope to have 100-150 participants. In addition to classes in the daytime, the evenings will have social events and plenary lectures. Bloomington is a wonderful place to visit, known for arts, music, and ethnic restaurants. All of this is within 15 minutes walking from campus. We aim to make NASSLLI fun and exciting.

Happy Truck Day!

• Leiter on King’ College London firing Shalom Lappin
• Shalom’s statement

The journal Nature Chemical Biology has an editorial that is well worth reading and pondering for other fields as well:

• 2010. On citing well. Nature Chemical Biology 6:2. p.79. doi:10.1038/nchembio.310.

Unfortunately, the editorial is behind a paywall. Here are the main points in excerpts.

The appearance of new ideas and discoveries in the scientific literature is a reflection of ongoing scientific progress. Individual articles are nodes of scientific knowledge, but citations of published work link together the concepts, technologies and advances that define scientific disciplines. Though information technology and databases have helped us to better manage the expanding scientific literature, the quality of our citation maps still hinges on the quality of the bibliographic information contained in each published paper. Because article citations are increasingly used as metrics of researcher productivity, the citation record also affects individual scientists and their institutions. As a result, all participants in the scientific publication process need to ensure that the citation network of the scientific literature is as complete and accurate as possible.

Many factors may stand in the way of good citation practices. [...] Each research group has its own referencing habits, and some may feature their own work too prominently or rely on familiar references without a critical examination of whether a particular citation is the most appropriate in the given context. Some researchers may not cite ‘old’ papers either because these are incorrectly viewed as being out of date or because inertia inevitably may encourage authors to cite the articles that show up more frequently in searches or that have appeared recently.

Researchers understandably are motivated, in both professional and personal ways, to have their scientific contributions recognized through citation by their peers. The community also values the accurate assignment of credit and precedence for scientific discoveries. As a result, even an accidental omission of a necessary citation may create an uncomfortable situation for a paper’s authors. More problematic, however, are cases where authors deliberately omit relevant citations. Because perceived novelty can be an important factor in determining where a manuscript is published, some authors may be tempted to avoid citing earlier or concurrent work from their own laboratories to enhance the apparent advance of a submitted study. In other cases, some authors may consider ongoing scientific disagreements, personal conflicts or competition a sufficient justification for omitting citations of work by others. Clearly authors need to do everything they can to avoid accidental omission of key references, and should never exclude relevant citations for nonscientific reasons. In turn, all scientists, independent of their roles as authors, referees or editors, need to renew their commitment to guaranteeing that literature citations correctly assign credit for ideas and discoveries and are placed thoughtfully in manuscripts and published papers.

Though editors and referees can help, authors are ultimately responsible for the information contained in their published papers. We recommend that authors take several important steps to increase the quality of their citation lists. First, principal investigators need to teach young scientists the appropriate ways to select manuscript references and mentor them in the ethical dimensions of citation. Second, authors need to put as much care into selecting and accurately citing references as they devote to the rest of their manuscripts. As part of this process, authors should perform comprehensive literature searches as they write and revise manuscripts, so as to identify relevant work that may need to be cited. Before including references in their citation lists, all authors should have read and discussed the candidate references to ensure that they are the most relevant choices and are called out at the appropriate point in the paper.

The responsibility for maintaining and enhancing the citation network of a discipline resides with all participants: authors, referees, editors and database managers. Thoughtful attention during the writing and review processes remains the first and best approach for ensuring citation quality and the appropriate assignment of credit in published papers. Yet new publishing and database tools that lead us to an interactive multidimensional scientific literature will become essential. As publishers move toward integrating functionality such as real-time commenting on published papers and creating ‘living manuscripts’ that preserve the snapshot of a research area through the lens of a published paper, while permitting forward and backward linking, the scientific literature is poised to become a richer environment that will support future scientific progress.

At S&P, we are fully committed to these goals, but could surely work harder to improve the citation practices. One of our criteria for evaluating submissions is “contextualization of research”, as spelled out in our inaugural editorial:

Are the main research questions contextualized in terms of earlier related work? Does the paper adequately cite related work? Could the impact of the paper be improved through modifications that would show the relevance of the results to future work in the same or other fields?

Advice to authors: by contextualizing results appropriately, the author not only increases the worth of the paper to the audience, but also makes the job of the editors and reviewers easier. It will be much easier for us to be sure that a paper should be published if we can clearly see what previous work it betters. Authors would do well to flag, both in the abstract and early on in the paper, the relationship of the paper to earlier proposals, and to indicate in broad terms what the relative advantages of the new approach are. Of course, it is then incumbent on the author to make sure that all such claims are fully justified in the main text of the article.

Another aspect of this is that once the relevant citations have been chosen, the bibliographic detail given in the article needs to be as complete and clear as possible:

• full first names of authors and editors
• both volume and issue numbers for journal articles
• page numbers for everything that appeared on numbered pages
• DOIs for every work that has a DOI (important both for easy access by readers to the cited literature and for all kinds of automated processes, present and future)
• URLs for unpublished manuscripts and other obscure sources
• conference proceedings formatted as specified in the Unified Style Sheet for Linguistics

S&P strives towards citing well, which requires continuous attention from authors, reviewers, and editors.

• Rick Nouwen: Two kinds of modified numerals, doi:10.3765/sp.3.3.

Keep on rolling.

Today marks the start of what we hope will be a banner year for our journal. We just published two articles:

• a main article by Sigrid Beck on “Quantifiers in than-clauses”
• a commentary by Uli Sauerland on “Embedded Implicatures and Experimental Constraints: A Reply to Geurts & Pouscoulous and Chemla”.

We have three more main articles in production, which should all appear quite soon:

• Donka Farkas & Henriëtte deSwart: “The semantics and pragmatics of plurals”
• Thony Gillies: “Iffiness”
• Rick Nouwen: “Two kinds of modified numerals”

We also have two or three more commentaries in various stages of submission/production, and are always soliciting commentaries on any of our main articles.

Finally, there are four papers under current review, we’re expecting revised versions of a number of manuscripts, and we’re awaiting several manuscripts that have been promised to us.

All in all, the journal is ramping up phenomenally and this will be the year that the quantity of our output will reach the levels of the other three main journals in our field.

Please help us make the journal more widely known and please submit your work to us. You’ll get excellent reviewing and editorial service and your work will look great thanks to our superior typography and it will be published free of charge and openly accessible to everyone.

The latest version is now available as a pre-print:

• von Fintel, Kai & Anthony S. Gillies. 2010. Must … stay … strong! URL http://mit.edu/fintel/fintel-gillies-2010-mss.pdf. Final pre-print, to appear in Natural Language Semantics.

We are grateful to NLS‘s reviewers and many colleagues who have given us useful feedback over the years (!) that we have been crafting this article.

Since the article is appearing in a Springer journal, we had to sign away our copyright. But MIT has negotiated fairly good terms with Springer that make their policies more or less compatible with our Open Access Policy. We are allowed to make the final pre-print openly accessible, including through MIT’s DSpace archive (where it will appear once the publisher’s version is out on the streets).