Science in the Open - Latest Comments

Re: What would scholarly communications look like if we invented it today?

psychemedia — Mon, 06 Sep 2010 18:49:41 -0000

Feel free to send me pointers to any feeds you get out of the lab, and I'll see what, if any, low hanging fruit ideas come to mind and try to hack something around for you ;-)

Re: Warning: Misusing the journal impact factor can damage your science!

psychemedia — Mon, 06 Sep 2010 18:47:14 -0000

I used to play with evolving classifier systems, back when i did "proper" research. One of the algorithms used to promote rules was a bucket brigade algorithm, that apportioned credit to rules that set up other rules that then received some sort of payoff.

Under this sort of approach, a coupled sets of questions are: what makes for tangible rewards and credit apportionment in research? Alternatively, do we need to come up with some other reputation frameworks? How would one based on pay-it-forward work, for example?

Re: What would scholarly communications look like if we invented it today?

CameronNeylon — Mon, 06 Sep 2010 15:53:13 -0000

Tony

I absolutely agree with all of this. One of the problems with the research
area is simply that most of the feeds aren't there at all though. Secondly
we have some real issues with what data standards we might use beyond things
like geo-tagging - but that shouldn't stop us trying! Certainly it would be
good to do more demonstrations with what is out there. I know that I've
fallen down in my mission to create more RSS feeds out of the lab and that
would be a good place to start, although we're starting to generate more of
our analysed data in feedable form. It would certainly be interesting to
think about what the low hanging fruit in the research space are and what
the best demos that could be put together are...

Re: What would scholarly communications look like if we invented it today?

psychemedia — Mon, 06 Sep 2010 05:25:50 -0000

I've been absolutely sold on the idea of feeds for ages, but the next step is to start growing out the ways in which they can be used. I've started gently exploring the idea of feed-oriented processing, trying to identify information processing design patterns that are generally useful and can be used as the basis of custom created, maybe even disposable, information processing tools or appliances. So for example, a Mendeley reading list powered current awareness tool, that spots journals on a reading list, and with an additional search term, then monitors the ToCs of the journals cited on the reading for articles containing the search term [ http://arcadiamashups.blogspot.com/2009/11/mashlib-pipes-tutorial-reading-list.html ]

A key part of the idea of feed oriented processing is that lists of content packages can flow around a network of interconnected applications using standard formats, and then processed in bulk. So for example, shove an RSS feed containing addresses in a description attribute through a tool that will geocode all the addresses and annotate each feed item accordingly; realising the feed now contains geo data, also express it as KML.

A partial network diagram on http://blog.ouseful.info/2010/08/25/my-slides-from-the-data-driven-journalism-round-table-ddj/ shows how different web applications can be wired together using URLs that represent outputs from other applications using standard format data published to a the URL.

The issue at the moment is educating folk- I'm trying to start with librarians - about the sorts of thing you can already do with pre-existing apps if content is published in a list form. The next step is showing them how, and getting them to build their own applications...

tony

Re: P ≠ NP and the future of peer review

CameronNeylon — Sun, 15 Aug 2010 02:59:52 -0000

Absolutely agree with the need for credit. This is a fundamental of getting anyone to do anything, and is at the root of the many cultural problems in modern research. What I find interesting is that credit can seem trivial, people work hard to get high kudos on Stack Overflow or to get Facebook/Friendfeed "likes" or retweets. As these become important to people they can start to become "real" (Stack Overflow scores have appeared on some CVs for instance). To some extent I see these high profile examples of community review as starting to provide some legitimacy for those kind of contributions precisely because they have a high profile.

In terms of the "technical quality of review" question. I've certainly heard (but can't lay my hands on any examples at the moment) CNS staff and editors say things like "the most rigorous peer review", "always send to reviewers at least twice for detailed comments", "often request extra experiments". Again this is partly confusion over what peer review does and what its for but the brands of CNS are bound up with the idea that quality and stringency of review is higher here.

But at some level I'd also invoke the "extraordinary claims require extraordinary evidence" rule. The examples I gave all made extraordinary claims. If peer review is going to do anything useful it must surely subject such claims to heavy duty scrutiny? I'm sure for every bad example exposed in a high profile journal there are many lower profile examples that never get picked up of course.

Re: P ≠ NP and the future of peer review

Becky — Sat, 14 Aug 2010 11:23:34 -0000

I think it's wrong to think of Science/Nature as aiming for a higher technical standard (or, at least, achieving it). They claim to distinguish themselves by two qualities, broad general interest and speed of publication. Speed is pretty much antithetical to highest possible technical review. One of the problems with the current situation, as you point out, is that the community has accepted "published in Science/Nature" as equivalent to "higher technical quality than the next paper", which is not (necessarily) the case.

I do understand that what you're talking about is substituting journal-driven peer review for community-driven peer review. My point is that peer review is real work, that substitutes for work you could be doing on your own papers, and it's hard enough to get good peer review when you apply a good deal of skill and effort to obtaining it.

You express a hope that the kind of community-based commenting and review that happens in mathematics will be eventually seen as valuable. This would be great. It would help if such community service were seen as valuable by search committees and promotion committees. Could we create a website where "the best public peer review" gets aggregated and credited? Then you could put it on your CV and get some benefit out of it.

Re: P ≠ NP and the future of peer review

CameronNeylon — Sat, 14 Aug 2010 08:32:49 -0000

I guess the point for me about the GWAS paper is that the criticisms in the 23andMe post as well as comments by dgmacarthur (I think it was) and others revealed fundamental technical issues that should have been exposed by technical review at any reputable journal, let alone Science where one presumes the standard is somehow higher than average (I can't speak to this myself as I've never reviewed for them nor had a paper get to review). It my be my reading of the post but it sounded as though there were fundamental technical checks that should be done for any GWAS paper that simply weren't.

In terms of the editor's problem, and I am an editor at PLoS ONE and for a new BMC journal soon to be launched, I couldn't agree more. The question is whether its worth the bother, is it more effective to simply put it out for anyone interested to review, and then allow the paper to be iteratively improved in response to those comments until it reaches some sort of level of agreement that it gets some sort of quality mark. I would argue that those papers that don't ever get a review should be released simply as data or blog posts or whatever. There will be some lost gems in there but that is a discovery problem - probably under the current system they wouldn't be published anyway.

Don't imagine that I'm saying peer review isn't useful. But the question, is which peers, and when, and how do you motivate them to contribute positively to the process of improving this piece of communication?

Re: P ≠ NP and the future of peer review

Becky — Fri, 13 Aug 2010 16:26:03 -0000

I completely agree about the problems. (And yes, Bayh-Dole was enormously counterproductive). I hope you're right that things are shifting. I don't think what happened with the GWAS paper is as significant as you imply, though — essentially this was a debunking of a high-profile paper, which would have happened anyway, the Web just made it faster.

I used to be a journal editor and in that role I spent a lot of time trying to identify reviewers who met the following characteristics: not a direct competitor, but knowledgeable in the field. Willing to spend a significant amount of time giving a paper a serious evaluation as a service to the scientific community. Not so busy that it will take months to get the report. Has good judgment. These are not easy criteria to meet, and it isn't hugely surprising if the system doesn't work perfectly. On the other hand, I worked at it, and I got quite a lot of useful reviews that visibly improved the papers I dealt with. I know this is anecdotal evidence but I think it may be wrong to dismiss peer review as not useful. It is a hard thing to do well, though.

Re: P ≠ NP and the future of peer review

CameronNeylon — Fri, 13 Aug 2010 10:23:22 -0000

Absolutely! That's partly what I was trying to argue here tho I'm not sure I made myself terribly clear. But I think as a community we need to take some responsibility for not demanding more as well. It's part of the avoidance of responsibility that I feel derives from the fact that as researchers we make the decision about where to (try to) send our papers but we don't have to deal with the cost, so we don't squeal and demand good value for money.

Re: P ≠ NP and the future of peer review

CameronNeylon — Fri, 13 Aug 2010 10:21:12 -0000

Becky, you won't get any argument from me on that. We know that this is hard and we certainly know that making a step change from one system to another won't happen. Equally tho, some of these pressures are going to shift pretty quickly. The patenting thing is certainly shifting in some quarters as people realise just how few of those patents are making any money at all, and how many of them are just blocks to innovation and opportunities for patent trolls.

Even on the issue of grant review I think things are shifting. Bottom line, organisations that judges individual people on the basis of the journal's they publish in (as opposed to the performance of their actual papers) need to take a long hard look at themselves because doing so is counterproductive and indeed highly non-scientific. You can't actually predict anything about the quality of a paper (within reasonable error bounds) from the journal it appears in.

The broader question, could this work in biomedical sciences? Well I would say we have some examples, the Science GWAS paper I mention being one. Combine that with the current focus on "broader impact" and a re-calibration of what publication means and I think we will see a gradual change. It will take time - and it may only ever be applied to high profile work - but I think things are shifting away from where we were a few years back and that's what I wanted to capture.

Re: P ≠ NP and the future of peer review

Becky — Fri, 13 Aug 2010 09:15:16 -0000

I agree that watching this happen has been wonderful. But I have no expectation that this method of reviewing will be practical in other fields, specifically biology. There aren't very many mathematicians in the world who can work at this level. None of them have to worry about their contribution to the proof being overlooked. Mathematics is (at least in principle) provably true or false, not a matter of interpretation. There is little room for cherished beliefs, and no ability to argue that the evidence must be wrong because it contradicts your view, or that it really proves your own hypothesis. There is no pressure from the university you work for to patent first before you share your results, and no chance of someone with a larger lab jumping on your early findings and overtaking you — having a large group doesn't actually help, in math. And mathematicians don't have to worry about the NIH reviewers concluding that their work must be unimportant because it's not published in the right journals. I am not arguing that the current state of peer review is good, just that there are reason to think that a transition to net review will not be easy.

Re: P ≠ NP and the future of peer review

bljog — Fri, 13 Aug 2010 04:19:48 -0000

Moving away from paper is indeed very difficult for scientists but I blame
the publishers. They make it harder for us by under investing in their
website infrastructure, not only in the tools available for commenting and
annotating but also with the lack of data available on their sites. Instead
of enriching the manuscripts we send them with the knowledge that is already
available on the web, they embed all the knowledge within a beautifully
formatted pdf that is too enticing not to print.
I'm hoping that tablets will change the way we work!

Re: P ≠ NP and the future of peer review

CameronNeylon — Thu, 12 Aug 2010 11:27:22 -0000

Fritz, thanks for your thoughtful comment. I think you're right on that there are two important characteristics here. One was the Deolalikar passed the initial "smell test" of credibility that meant serious people would take a look, and secondly as you say that Lipton was a key moderator. I've written before (tho not explicitly) about how we need to build social infrastructure that supports people like Lipton to mediate and support the application of expert attention. These people are "market makers" of a sort who provide the credibility that there is sufficient "capital" of attention to make a contribution worthwhile. These are very special people but they are often not held in high regard by people outside of their immediate circle.

In my mind I can imagine a sort of technical solution that provides a place where these people can mediate interactions at lots of different levels of specialisation and with different levels of reputation. We could probably build such a thing (indeed StackOverflow is arguably precisely such a thing for programmers) but it won't work unless there is significant uptake and validation of the work of the mediators as an important contribution to the research community. So absolutely the key to this is understanding the sociology of this as a route to making it happen effectively more often.

Re: P ≠ NP and the future of peer review

CameronNeylon — Thu, 12 Aug 2010 11:14:39 -0000

I agree there are technical issues and usability issues with current commenting systems but even with good systems you see limited comments. There simply aren't that many of us interacting with papers online. If you look at number of bookmarks in Citeulike/Mendeley or Delicious the numbers are pretty low. Most people land on the html, download the pdf, print and then annotate by hand. Nothing wrong with that, its a good workflow, but until we see more of the interaction happening in online spaces then the technical problems are relatively small.

Now someone did do a pubmed-Disqus mashup that looked really interesting, but I can't find the link at the moment. Looked great, worked well, but are we using it? Even me?

Re: P ≠ NP and the future of peer review

Fritz Juhnke — Thu, 12 Aug 2010 04:41:36 -0000

Thank you for this post. I am as fascinated as you are by the process of peer review that has unfolded. It is the sort of phenomenon to restore my faith in human nature. I have been feeling warm and fuzzy for days because of it.

You have recognized that something very important is going on. I hope that others realize it as well. However, for anyone who wants to see this kind of thing happening more often, indeed to actively encourage it to happen more often, it is important to understand *why* it is happening. Yes, part of the reason is the extreme importance of the topic. Yes, part of the reason is that the Internet exists. But these are far from sufficient reasons. One only needs to consider what happened when the first million-dollar Millenium Prize problem was solved in 2003. That was a parallel situation, with a purported but incomplete proof distributed over the Internet only. In that case, however, the verification of the proof was fractured, prolonged, and bitter. Mathematicians don't always cooperate this well. If you don't know that story, you may read about it here: http://www.newyorker.com/archive/2006/08/28/060828fa_fact2

Let me add my hunch as to what happened differently in this case. I have seen collaborative phenomena such as the current one unfold in a few widely different contexts, as well as seen it die in infancy in many, many others. In my observation, the point of critical importance to whether teamwork happens or fails to happen is the quality of the moderator. I conjecture that it was no accident that the P=NP discussion crystallized around Dick Lipton's blog as opposed to somewhere else, and furthermore that absent Dick Lipton, it wouldn't have happened at all. Experts would have made scattered observations in various blogs and e-mails, collectively producing less in three weeks than the working group produced in three days.

Most such discussions are smothered in the cradle because the people whose contributions are most valuable choose not to contribute. The discussion isn't killed internally, it dies from the best people staying away in droves. Their reasons for absence include fear of not being listened to, fear of being drowned out by idiots, expectation that the discussion will have no result, and fear of not being credited for their contributions if a positive result is produced. Yet anyone may rapidly verify from his blog that Lipton is generous with his own ideas, respectful of others' ideas, fair-minded, competent to judge worthy from unworthy contributions, and eager to give credit where credit is due.

Like you, I would love to see the whole world work like the peer review of Deolalikar’s proof. To this end I believe we must understand it as a social phenomenon held together by the glue of an extraordinary individual. Lipton is exceptional not by virtue of being the most brilliant mathematical mind applied to the problem, but by having the rare social wisdom necessary to bring other great minds together.

I hope some sociologist picks up the ball for understanding such cooperative phenomenon better, because a society that comprehends it properly will be great above all societies.

Re: P ≠ NP and the future of peer review

bljog — Wed, 11 Aug 2010 09:51:47 -0000

I think that one reason the commenting and "likes" etc post-review process hasn't taken off is that the infrastructure for making those comments is poor. As a researcher reviewing online, you would want a system that gives you credit for the work you are doing, also somewhere you can go to see all the comments and reviews you have made on different articles across different journals and blogs. You would want to be in control of those reviews. I have made a number of comments both on PLoS and Biomed Central but there is no way of me accessing, reviewing or deleting those comments I made and have never received any comments back from authors. If a system like DISQUS was set-up across different journals/publishers, you would have a way of being in control of the work you have done and be able to build a reputation in the research community by making reviews/comments and getting liked or disliked for it.

Re: The triumph of document layout and the demise of Google Wave

Robertm — Mon, 09 Aug 2010 18:32:20 -0000

It is fairly straightforward to configure a server on which MathJax is hosted so that it can be used from pages hosted in another domain. I understand not everyone would be willing or able to do this on a server, but the installation at mathjax.org has been configured in this way precisely to support people using Blogger. See https://sourceforge.net/projects/mathjax/forums/forum/948701/topic/3747270 for details.

Re: The triumph of document layout and the demise of Google Wave

cgranade — Sat, 07 Aug 2010 13:47:36 -0000

I love jsMath and MathJax, but there's a peculiar problem I've run into with using them. Unless I'm terribly mistaken, both require a few files to be placed on the same domain as a page using the toolkits. This makes it hard to use them with, say, Blogger or WordPress (running on a shared server, anyway). Since blogs form an important channel for discussion, I would have hoped that this problem would be tractable, but so far, I've not run into a good solution. That said, I wholeheartedly agree that such toolkits are a good way forward, as they allow for LaTeX to be used as an intermediate, author-facing language, and yet retain the benefits of standards-compliant web technologies.

Re: The triumph of document layout and the demise of Google Wave

crusbrid — Fri, 06 Aug 2010 11:38:56 -0000

I've just read Brian Kelly's post on formats for repositories, in which he talks about the EPUB format (see http://ukwebfocus.wordpress.com/2010/08/04/epub-format-for-papers-in-repositories/ for the post). He says:

'EPub is described in Wikipedia as “a free and open e-book standard by the International Digital Publishing Forum (IDPF)“. The article goes on to add that “EPUB is designed for reflowable content, meaning that the text display can be optimized for the particular display device used by the reader of the EPUB-formatted book. The format is meant to function as a single format that publishers and conversion houses can use in-house, as well as for distribution and sale.“

'In terms of the open standards used EPub consists of three specifications:

'Open Publication Structure (OPS) 2.0, contains the formatting of its content.
Open Packaging Format (OPF) 2.0, describes the structure of the .epub file in XML.
OEBPS Container Format (OCF) 1.0, collects all files as a ZIP archive.

'The articles states that “EPUB internally uses XHTML or DTBook (an XML standard provided by the DAISY Consortium) to represent the text and structure of the content document and a subset of CSS to provide layout and formatting. XML is used to create the document manifest, table of contents, and EPUB metadata. Finally, the files are bundled in a zip file as a packaging format.“'

It sounds much more "of the web" than PDF, and might be very interesting!

Re: The triumph of document layout and the demise of Google Wave

crusbrid — Fri, 06 Aug 2010 11:07:55 -0000

Cameron, I'm not quite sure I "get" the theme of this post, but I'll confine myself for a minute to Utopia Document. What bothers me here is the very strongly different nature of the tool. Compare this paper, for example: Ruthensteiner, B., & Hess, M. (2008). Embedding 3D models of biological specimens in PDF publications. Microscopy research and technique, 71(11), 778-86. doi: 10.1002/jemt.20618. This doesn't require a different viewer, but let's you view the models in PDF. It seems to work quite well for me.

I'm keen in principle on the idea of scientific articles migrating from PDF as default to HTML as default. In theory, with a little added RDF etc, we could do wonderful things, embedding supplementary data. However, in practice the result is pretty much rubbish even for quite ordinary articles. It seems to be very hard to get figures and tables to look right and be readable in context. And worse, there doesn't seem to be a standardised way of saving a packaged web page so that it can be left for years and then read even when the originating site has gone. Safari's .webarchive format is able to do that, but is not readable by anything else as far as I can see. the standard "save html" format seems to be a .html file and an associated directory of other stuff; hardly handy, even if in practice several browsers seem to be able to read it!

Re: The triumph of document layout and the demise of Google Wave

mleifer — Fri, 06 Aug 2010 10:06:08 -0000

As someone who has spent a bit of time trying to write LaTeX maths plugins for various webapps, I feel a lot of frustration about this. The tools to do this extremely well have been around for a fairly long time, but most of them are directed towards converting complete LaTeX documents into standalone webpages and it is not easy to adapt them to plug into other systems, especially if you are trying to maintain compatibility with basic webhosting accounts. As a result, almost all LaTeX plugins known to me are hacks and they usually have fairly serious bugs because people only test them on a few simple equations rather than writing out long mathematical expositions.

The few systems that do handle mathematics very well are usually built with the idea of mathematical formulas being first class objects from the outset, rather than trying to build maths support into an existing system built by people who have no idea about the needs of mathematical scientists. One of the best I know of is http://cnx.org/ which will let authors submit LaTeX and convert it into very good HTML+MathML automatically. Basically, I think we should build more systems like this, i.e. wiki and blog engines with ground level maths support, rather than trying to hack existing systems.

On the rendering front, I am hopeful that adoption of MathJax http://www.mathjax.org/ will greatly improve the situation. This is not an issue that we can sit back and wait for the browser developers to deal with, since they have clearly demonstrated that MathML support is a marginal concern even for those browsers that proudly tout their "standards compliant" status. The situation is rather more like that of javascript, where people are resigned to browser inconsistencies, but handle it via libraries like jquery rather than waiting for the browser developers to do something about it.

Re: The triumph of document layout and the demise of Google Wave

CameronNeylon — Fri, 06 Aug 2010 06:45:35 -0000

Indeed, maths is a serious problem on the web, and something that has definitely made people very keen on TeX. Again though my perception is that alternatives have not got traction because they are not seen as being as "pretty" as TeX. Getting TeX/MathML support into Wave was something that @axiomsofchoice and others put quite a bit of effort into and there were some nice examples but they were proof of concept not ready for the mainstream.

My feeling is that what we will see is systems that enable TeX based authoring of maths which then get converted to MathML (which is a semantics and layout tool really and not much good for authoring) and displayed nicely but flexibly. The first bit exists but I'm not sure that the rendering is there yet. Ultimately it will probably be possible to just directly write the maths down and have it recognised and converted to MathML

Re: The triumph of document layout and the demise of Google Wave

cgranade — Thu, 05 Aug 2010 19:58:43 -0000

I see a few things holding back web-based alternatives to the paper model, actually. One of the most frustrating to me has to be the complete lack of widespread math support as extensible and as mature as LaTeX. I feel like having LaTeX-grade math support in a web-based platform would do a lot for adoption, as it would broaden the range of fields that could benefit rather drastically.

Sure, there's some LaTeX support mixed here and there in blog engines, Wikis and similar, but from what I've seen, they tend to suffer a lack of extensibility and rely on bitmap images that don't rescale properly with the text, much less fit in typographically. MathML was supposed to deliver us from this, but between half-assed browser support and an overly verbose syntax that is unsuitable for "by-hand" use, it seems to have fizzled by comparison to LaTeX-based solutions.

A distraction from Wave, to be sure, but I think that the document model will persist in the sciences at least as long as LaTeX has a monopoly on the typesetting of mathematics, frustrating implementation of Wave-like solutions.

Re: The Nature of Science Blog Networks

CameronNeylon — Mon, 26 Jul 2010 05:58:55 -0000

Absolutely! Coffee and in person are good as well!

Re: The Nature of Science Blog Networks

Eva — Mon, 26 Jul 2010 05:49:06 -0000

Great post. Insightful as usual. Instead of leaving a comment, can I comment in person at SciFoo over coffee? I know you like everything to be online and accessible, but I like coffee and meeting in person. Ha!