Disruptive Science: Brave New WorldViews

Current Biomedical Publication System: A Distorted View of the Reality of Scientific Data?

2008-10-13T13:00:00.012-05:00

Why Current Publication Practices May Distort Science

Young NS, Ioannidis JPA, Al-Ubaydli O

PLoS Medicine Vol. 5, No. 10, e201 / October 7 2008

[doi:10.1371/journal.pmed.0050201]

Summary

The current system of publication in biomedical research provides a distorted view of the reality of scientific data that are generated in the laboratory and clinic. This system can be studied by applying principles from the field of economics. The “winner's curse,” a more general statement of publication bias, suggests that the small proportion of results chosen for publication are unrepresentative of scientists' repeated samplings of the real world.

The self-correcting mechanism in science is retarded by the extreme imbalance between the abundance of supply (the output of basic science laboratories and clinical investigations) and the increasingly limited venues for publication (journals with sufficiently high impact). This system would be expected intrinsically to lead to the misallocation of resources. The scarcity of available outlets is artificial, based on the costs of printing in an electronic age and a belief that selectivity is equivalent to quality.

Science is subject to great uncertainty: we cannot be confident now which efforts will ultimately yield worthwhile achievements. However, the current system abdicates to a small number of intermediates an authoritative prescience to anticipate a highly unpredictable future. In considering society's expectations and our own goals as scientists, we believe that there is a moral imperative to reconsider how scientific data are judged and disseminated.

Full Text Available At:

[http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0050201]

Box 1. Potential Competing or Complementary Options and Solutions for Scientific Publication

Accept the current system as having evolved to be the optimal solution to complex and competing problems.

Promote rapid, digital publication of all articles that contain no flaws, irrespective of perceived “importance”.

Adopt preferred publication of negative over positive results; require very demanding reproducibility criteria before publishing positive results.

Select articles for publication in highly visible venues based on the quality of study methods, their rigorous implementation, and astute interpretation, irrespective of results.

Adopt formal post-publication downward adjustment of claims of papers published in prestigious journals.

Modify current practice to elevate and incorporate more expansive data to accompany print articles or to be accessible in attractive formats associated with high-quality journals: combine the “magazine” and “archive” roles of journals.

Promote critical reviews, digests, and summaries of the large amounts of biomedical data now generated.

Offer disincentives to herding and incentives for truly independent, novel, or heuristic scientific work.

Recognise explicitly and respond to the branding role of journal publication in career development and funding decisions.

Modulate publication practices based on empirical research, which might address correlates of long-term successful outcomes (such as reproducibility, applicability, opening new avenues) of published papers.

[http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0050201#journal-pmed-0050201-box002]

>>>Extended Version<<<

The Market for Exchange of Scientific Information: The Winner’s Curse, Artificial Scarcity, and Uncertainty in Biomedical Publication

http://medicine.plosjournals.org/archive/1549-1676/5/10/supinfo/10.1371_journal.pmed.0050201.sd001.pdf

Guest Blog

More Evidence on Why We Need Radical Reform of Science Publishing / Richard Smith

PLoS Medicine invited Richard Smith, former editor of the BMJ and current board member of PLoS, to discuss an essay published this week by Neal Young, John Ioannidis and Omar Al-Ubaydli that argues that the current system of publication in biomedical research provides a distorted view of the reality of scientific data.

"For me this paper simply adds to the growing evidence and argument that we need radical reform of how we publish science. I foresee rapid publication of studies that include full datasets and the software used to manipulate them without prepublication peer review onto a large open access database that can be searched and mined. Instead of a few studies receiving disproportionate attention we will depend more on the systematic reviews that will be updated rapidly (and perhaps automatically) as new results appear."

[http://www.plos.org/cms/node/409]

News Coverage

The Economist [10-09-08] : Scientific Journals: Publish and Be Wrong

"Dr Ioannidis made a splash three years ago by arguing, quite convincingly, that most published scientific research is wrong. Now, along with Neal Young of the National Institutes of Health in Maryland and Omar Al-Ubaydli, an economist at George Mason University in Fairfax, Virginia, he suggests why."

>>>With Comments<<<

[http://www.economist.com/science/displaystory.cfm?story_id=12376658]

Newsweek [10-06-08] : Don't Believe What You Read, Redux / Sharon Begley"

Bottom line: when it comes to 'the latest studies,' take what you read with a grain of salt.

[http://www.blog.newsweek.com/blogs/labnotes/archive/2008/10/06/don-t-believe-what-you-read-redux.aspx]

Related

Why Most Published Research Findings Are False / John P. A. Ioannidis / PLoS Med. 2005 August; 2(8): e124 / August 30 2005

[doi:10.1371/journal.pmed.0020124]

Summary

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.

Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

[http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124]

CHE: Rebels, Mavericks, and Heretics in Biology

2008-07-10T12:14:00.019-05:00

Dynamic Dissent
Iconoclasts, both visionary and misguided, have advanced modern biology

By OREN HARMAN and MICHAEL R. DIETRICH

The history of science is invariably told through the prism of its heroes, and modern biology is no exception. Through those heroes, historians have analyzed the growth of the life sciences, the rise of their institutions and subfields, and the evolution of biologists' understanding of nature. But these histories are not the whole story.

The story of biology can also be told through the gaze of its rebels: those men and women who challenged the prevailing pictures of life. Some of these rebels have been called cranks, others gadflies, still others prophetic. Some of them were in fact wrong; others, though lambasted for their views at the time, will be found — or have already been found — to deserve more-just treatment. The value of studying those who disagreed does not lie in the vindication of previously maligned concepts, theories, or findings. Rather, scientific mavericks have much to teach the challenged about themselves, about the value of "fruitful errors," and about the many paths to discovery and innovation in science.

[snip]

In the just-published book Rebels, Mavericks, and Heretics in Biology, we commissioned essays on 19 of the most notable iconoclasts in the last 150 years of biological research. Our goal was to understand the dynamics of dissent as we studied how those biologists sought to topple scientific icons ... .

[snip]

One of the first generalizations that emerges from these historical cases of rebellion and innovation is how often iconoclasm results when scientists transgress disciplinary boundaries.

[snip]

Carl Woese's discovery in the late 1970s of new kingdoms of microbial life was a product of his synthesis of microbiology and molecular evolutionary biology. Woese sought to solve a problem that more-conservative microbiologists believed didn't even exist (they assumed bacteria constituted a single kingdom). The historian Jan Sapp's reflection on Woese and his work captures the transdisciplinary effect, when he writes about the "saltational effects of the lateral transfer of concepts and techniques between fields": "Small changes, refinements, occur within disciplines; large-scale changes may result from the sharing of innovations between them."

[snip]

At first glance, iconoclasm conjures images of research that is novel, pathbreaking, disruptive, and forward-looking. Indeed, we found several biologists who fit that description well. Barbara McClintock's work on transposable genetic elements and Roger Sperry's research on the split brain were initially considered too novel, were actively resisted, and eventually earned McClintock and Sperry Nobel Prizes.

[snip]

Between the two extremes of reaction and innovation, an interesting designation emerges. For many of the "rebels" in biology, there is a sense that they are both simultaneously behind and ahead of their times. Darlington harked back to a theory-based outlook exemplified by 19th-century giants such as Weismann, and which, although considered "unscientific" by peers, would be picked up again in the future by theoreticians and experimentalists trying to understand the role of dynamic genetic systems in evolution. McClintock and Goldschmidt, too, tied development and heredity to evolution in the 19th-century tradition, and, though they were frowned at by contemporaries, actually pointed to problems that would be dealt with only decades into the future.

With respect to the scientific rebel, then, a Tolstoyan thesis emerges: While all conventional practitioners in the life sciences may be said to be conventional in the same way, all rebels seem to rebel in their own particular fashion. ... Singling out the rebel helps us understand how and why we come to our beliefs and practices, and how and why we struggle to hold on to them.

Raymond Arthur Dart, the man who discovered Australopithecus africanus in 1924, is a good example. While the evidence he unearthed seemed to strongly suggest an African origin of mankind, it took almost two generations to adopt that revolution in our understanding of human evolution. As Dart's student Phillip Tobias shows in our book, that lag had more to do with the political resistance to such an admission than with scientific evidence.

[snip]

Barbara McClintock, the famed geneticist, may have helped create her own image as a rebellious personality fighting against the odds of a marginalizing scientific community not quite able to appreciate the brilliance of her own, bucking scientific claims.

[snip]

If one single quality unites all the scientists we feature in our book, it is stubbornness, their steadfastness in their challenges to orthodox thought. George Bernard Shaw wrote: "The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." That may or may not be true in science; it would be a stretch to call all iconoclastic biologists "unreasonable," and it is unclear how some iconoclasms have benefited mankind.

[snip]

The issue of scientific legitimacy cannot rest on the simple dichotomy between what ends up being "right" versus what ends up being "wrong."[snip]

Telling the stories of biology's rebels, mavericks, and heretics is an exercise in neither vindication nor rehabilitation. These stories reveal that the history of biology is also a history of dissent.[snip] Focusing on the moments of dissent, however, provides a valuable contrast to those ideas and biologists currently considered to be correct. Instead of a seamless narrative of progress or even of heroic struggle to find the truth, we get a history marked by innovation, resistance, and defiant theories and personalities. [snip]

Oren Harman is an assistant professor in the graduate program for science, technology, and society at Bar-Ilan University (Israel), and Michael R. Die-trich is an associate professor of the history and philosophy of biology at Dartmouth College. This essay is adapted from a book they edited, Rebels, Mavericks, and Heretics in Biology, just published by Yale University
Press.

Source

The Chronicle Review / Volume 54 / Issue 44 / Page B7 /

[http://chronicle.com/cgi-bin/printable.cgi?article=http://chronicle.com/weekly/v54/i44/44b00701.htm]

Book

Rebels, Mavericks, and Heretics in Biology / Edited and with an Introduction by Oren Harman and Michael R. Dietrich and with an Epilogue by R.C. Lewonti /May 19, 2008 / 416 p. / ISBN: 9780300116397 ; ISBN-10: 030011639X /

[http://yalepress.yale.edu/book.asp?isbn=9780300116397]

Scientists featured in this volume:

Alfred Russel Wallace
Hans Driesch
Wilhelm Johannsen
Raymond Arthur Dart
C. D. Darlington
Richard Goldschmidt
Barbara McClintock
Oswald T. Avery
Roger Sperry
Leon Croizat
Vero Copner Wynne-Edwards
Peter Mitchell
Howard Temin
Motoo Kimura
William D. Hamilton
Carl Woese
Stephen Jay Gould
Thelma Rowell
Daniel S. Simberloff

CfP: Science of Science: Conceptualizations and Models of Science

2008-05-13T13:46:00.012-05:00

Call for Papers Call for Papers: Journal of Informetrics 3(1) / January 2009

Special Issue on “Science of Science: Conceptualizations and Models of Science”

Guest Editors: Katy Borner, Indiana University & Andrea Scharnhorst, Royal Netherlands Academy of Arts and Sciences

This special issue aims to improve our understanding of the structure and evolution of science by reviewing and advancing existing conceptualizations and models of scholarly activity.

Existing conceptualizations and models of science have been created by scholars from very different disciplines and backgrounds. They have the form of

philosophical concepts (Bernal, Kuhn, Popper),
(utopian) stories (Wells, Lem),
visual drawings (Otlet),
empirical measurements (Price, Garfield), or
mathematical theories (Goffman, Yablonski)

among others.

It is our belief that a theoretically grounded and practically useful shared conceptualization of science can provide the intellectual framework to interlink and puzzle together the hundreds of science models in existence today. This is analogous to how meteorologists or seismologists integrate rather different local weather models or seismic hazard predictions into a global coherent model that has higher predictive value and broader coverage. With this issue we aim to start an interdisciplinary discourse towards a science of science models.

The design of such a conceptualization requires the identification of the

Boundaries of the system or object.
Basic building blocks of science, e.g., units of analysis or key actors.
Interactions of building blocks, e.g., via coupled networks.
Basic mechanisms of growth and change.
Existing laws (static and dynamic).

Ideally, the conceptualizations can be also presented in a visual form so that disciplinary and cultural boundaries can be bridged more easily.

This issue invites contributions such as

Reviews of existing conceptualizations of the structure and evolution of science. Each paper should compare and contrast works from multiple authors. Here, we invite contributions by philosophers, sociologists and historians of science as well as scientometricians.

Historiographic and ethnographic work on how people understand and communicate the structure and dynamics of science via imagery and textual descriptions. Papers in this category should analyze a variety of approaches, including critiques on science conceptualizations.

Novel conceptualizations and empirically validated models of science and scientific communication. Please discuss epistemic assumptions and disciplinary roots, possible application domains, covered and omitted features of scientific evolution, and model interpretation. Work on ‘ensemble models’ that integrate different mathematical models to arrive at higher quality and broader coverage simulations of science are welcome.

Authors are also welcome to discuss alternative paper proposals with the guest editors.

***Deadlines***
Submission of 2-page abstracts: May 30th, 2008
Submission of full papers: Aug 31st, 2008
Reviews back and accepted papers shared: Oct 31st, 2008
Final version due: Nov 30th, 2008

Please send the abstracts to

Katy Boerner

Victor H. Yngve Associate Professor of Information Science, Director of the Cyberinfrastructure for Network Science Center

School of Library and Information Science / Indiana University / 10th Street & Jordan Avenue / Wells Library 021 / Bloomington, IN 47405, USA

katy@indiana.edu / Phone: 812 855 3256 / Fax: 812 855 6166

[http://ella.slis.indiana.edu/~katy/]

SOURCE

[http://cns.slis.indiana.edu/news/Informetrics_Science_of_Science.pdf]

Launch of Disruptive Science: Brave New WorldViews

2008-05-11T12:44:00.037-05:00

Friends/

Science Has A Future That Will Be Different From Its Past That Will Be Different From Its Present.

Disruptive Science: Brave New WorldViews is devoted to documenting research and speculation that present alternative explanations and minority views about The Natural and Physical World.

While its focus is on plausible alternative perspectives, it will also give serious consideration to serious explanations that may not currently be accepted by the academic and scientific communities.

The Disruptive Science blog was formally established on Mother's Day 2008.

/Gerry