I am reading up a lot of Jim Gray these days, so a lot of his ideas are quite fresh in my head. Also had an interesting discussion with Nancy Parmalee on Twitter about software, informatics, bench scientists and small labs. One thing that jumped out, and is hardly a surprise, is that for the most part, there is a huge disconnect between the data science, and the scientists who need to make use of the work done by data scientists (often bench scientists). I’ve long argued that we neglect “infrastructure” software like data management systems, tracking systems, query systems, etc which all require well designed, scalable backends and should be treated like products, cause they are, even if they are home grown, or derived from open source software.

This brings me to Jim Gray and his “rule of 20″ (came in the context of the Sloan Digital Sky Survey and discussed in the Fourth Paradigm). He argued that for the database builder, the researchers querying the data were asked about the 20 questions they most wanted answers to. Five would be too few, and a hundred would become too general, and systems are usually better built for a purpose than become too general purpose. I’ve seen this process work, and indeed participated in similar processes. One of the points Nancy brought up was the lack of a common language. This is how you arrive at common semantics. Also important in this discussion is the data scientist. While they can maintain more general purpose systems for ad hoc querying and data mining (e.g. just flat namespaces to be queried with mapreduce), they are the providers of much of the information that those twenty questions will address, so it becomes a three stakeholder game and you need to approach this in an iterative manner, constantly refining the process.

Will we ever get there in an academic setting? I am not sure, but in the places I have seen that process in action, it seems to work. We live in an era of constant change and by creating artificial barriers and ignoring things we can do at the software layer, we’re only making things more difficult for ourselves.

Related articles by Zemanta

• Data science, roles, and barriers (mndoci.com)

click for full size

In this slide (the narrative can be found in the Fourth Paradigm), Jim states a belief I share. That the internet can do a lot more than just make the text of research papers available. The whole point of the Web as a platform for scientific discovery is lost until we can unify scientific data with literature in the sort of pyramid shown above. To be able to jump between paper, data, commentary, reviews, further analysis, etc. The interesting concept Jim brings up is overlay journals. An overlay journal is one where you have data archives and literature archives with appropriate content going into the appropriate place (whether it’s actual context or a distributed index is a different story). One should, in theory, with the raw data, appropriate APIs etc, be able to construct a meta-journal around an interest area (The collections on Nature Precedings, and similar efforts at PLoS come to mind). Perhaps we could do with pre-publication repositories, or create loosely coupled peer-reviewed journals. I’d like to see publishers and web science types (Timo?) push the envelope here.

Related articles by Zemanta

• Collections on Nature Precedings (mndoci.com)
• Hey maybe scientists should do more than just wait for their journal to issue a press release on their new fabu article (scienceblogs.com)
• Peer-review has a place (mndoci.com)

Tyler Cowen has an different viewpoint on this subject (he is a tenured professor). I don’t know if that was the intent, but you get the impression that he doesn’t quite like the idea, but he brings up a great point. No one has suggested a model that would replace it and there is no real debate cause the counterpoint is mostly against tenure without a solid alternative.

Steven Levitt doesn’t like tenure much either

I’ve never been comfortable with the idea of tenure, but don’t have enough evidence that it’s a bad thing per se. Tenure isn’t exactly easy to come by, especially at quality universities, so there is still work required to get there. However, coupled with the other incentive systems in place for researchers, you end up with a rather vicious publish or perish system.  So you end up in a system which, from a naive point of view, hampers creativity and risk taking and in the end probably limits what problems we try and solve and how.  Of course, you could argue that you get a pulpit from which you can do some really groundbreaking work, but I think that argument breaks down because your funding comes from a different source.

What do you think? Is tenure a good thing? Is it the best in the absence of a viable alternative? Do you consider it the pinnacle of your career, or just another step?

Related articles by Zemanta

• What if universities get rid of tenure? (marginalrevolution.com)
• Tenure: An Idea Whose Time Has Gone (theatlantic.com)
• How Much Does Tenure Cost? (theatlantic.com)
• The Steady Disappearance Of Tenure (camelswithhammers.com)
• A Call to End Teacher Tenure (freakonomics.blogs.nytimes.com)
• What’s So Horrible About University Tenure? (businessinsider.com)

in an information-rich world the wealth of information means … a scarcity of whatever it is that information consumes … the attention of the recipients

One of the challenges we face in the sciences is measuring the relevance of a particular piece of research or a particular researcher. From measures that are just poor representations of individual papers or even people (yes Impact Factor I am looking at you), to naive yet well intentioned steps in the right direction (Article Level Metrics) there are a lot of attempts to try and determine relevance and influence in scientific publishing. Personally I still like Eigenfactor a lot, but there is still room for improvement.

As I read through Daniel’s slides, it was apparent to me that while his area of research is quite different, the underlying principles of authority and scarcity are universal and apply really well to scientific publishing, and in fact his work shows exactly why Impact Factor is a bad metric, since it only reflects the average of citations to articles published. Specifically does not do so at a granularity where it is really useful and there is limited context. Science is not much about gaming the system, but the IF is easy to game. in fact if an author and his own students who go on to do their research keep citing each other, they are essentially doing that implicitly. See slide #27 in the slides embedded below.

Anyway, for those of you into citation metrics and interested in objective methods of scientific authority and relevance check out these slides

Related articles by Zemanta

• A Better Way To Filter Twitter’s Spambots? Ask Google (blogs.forbes.com)
• Metrics: Do metrics matter? (nature.com)
• Post-Publication Review: Does It Add Anything New and Useful? (scholarlykitchen.sspnet.org)
• Metrics: A profusion of measures (nature.com)

This is a big deal. I am not quite yet sure how the integration will happen and how Freebase data will start surfacing in Google results. The key for Google is to keep Freebase up and running and figure out how to pull that information into the core search function at Google. It is also entirely possible that this was a play for the excellent Freebase team, although that wouldn’t make complete sense, given the powerful platform and tools that the Freebase gang has created. It’s going to be fun to see how this evolves.

Related articles by Zemanta

• Google Tries Freebasing (GOOG, GS) (businessinsider.com)
• Google Gets Semantic: Buys Metaweb (gigaom.com)

After working in companies both large and small, it’s clear to me that the strict separation of tasks is the major obstacle faced by data scientists. The most common manifestation is the separation between data analysis and data management. In many large companies, most analysts/statisticians have to wait for data from a designated data warehousing team, and in a lot of cases they wait for data from multiple owners of different data warehouses.

Neil pointed to another bit

To nurture data scientists, companies need to focus more on culture and organizational structure. Many data workers have enough skills and training to quickly become productive in multiple areas of data intelligence. The problem is that most don’t work in environments that encourage them to become data scientists. They’re stuck in silos and limited to one or two areas of data intelligence. Often, they’re restricted to use tools “approved” by their managers.

When I wrote about the biological data scientists, one of the points I was trying to make was that advanced data analysis requires a set of skills that are not easily transferable to bench scientists or biologists who want to retrieve specific, limited, pieces of informations. While at a particular point in time different people in a company or lab might focus on different aspects of the task, a data scientist needs to be capable of carrying out different aspects of the task. Ben covers these different aspects as well.

• data acquisition: this might entail writing custom parsers and web crawlers, or scripts that target specific web services or API’s for non-traditional data sources.
• data management: ETL, manipulate, query, and maintain data in databases, key-value stores, or Hadoop.
• information visualization: uncovering patterns through the use of static visualization toolkits and/or interactive platforms based on Flash or Javascript.
• analytics: this can range from simple to complex techniques in multivariate statistics, machine-learning, and NLP.
• insight: extract, summarize, and present key findings to a broad audience.

We need to rethink how we approach data as bioinformaticians. It will not be acceptable to be considered a bioinformatician with skills that don’t cover at least a good chunk of the skills described above. But to get there, we need to change our idea of the role of bioinformaticians and of the value of the large volumes of data we are generating.

Related articles by Zemanta

• The Biological Data Scientist (mndoci.com)
• Data-driven research products (mndoci.com)

• Mailing lists and newsgroups: The first form of Q&A that I ever used was a mailing list My PhD thesis calls out the Computational Chemistry List and the CHARMM mailing list for helping me learn a lot about the tools and science I needed to learn to successfully graduate. Closely related, since they are often a hybrid, are newsgroups, with Google Groups probably being the best known today
• Forums: Probably the most common form of Q&A in the technical world today. It’s how support is provided, and sites like SEQanswers are a treasure trove of information. I can safely say that a lot of what I’ve learnt about writing and producing electronic music came from the forums at KvR. The CHARMM mailing list has long been a forum
• StackOverflow: It’s safe to say that Stackoverflow has been a resounding success, if for no other reason than the number of sister sites, and Stackoverflow-style sites have come up. From Biostar to the various metaoptimize sites (Scale, ML/NLP), a plethora of sites have come up which use the reputation model and design style of Stackoverflow. SEQanswers is also experimenting with a similar site, and in general this model seems to do better for specific topics
• NLP driven: Sites like Quora (which to me is different from other, more general sites like Aarvark and Yahoo Answers) approach the technology Q&A problem differently. Quora takesideas from Stackoverflow, others from Aardvark, and also from sites like Mahalo and Wikipedia, and distills them into a model that has yet to be proven, but shows a lot of promise. It will be interesting to see how Quora evolves. Like Stackoverflow it looks designed to Google well.
• More general sites: Here I include sites like Hacker News, Friendfeed, etc. On Hacker News, you will often fine hackers asking specific questions, and the rooms on Friendfeed, like The Life Scientists, are a great place to ask questions. And I haven’t even talked about IRC.

Each approach has pros and cons, and quite honestly, at some level I like all of them, although mailing lists and newsgroups are less usable and useful for me with the exception of a few very specialized, low volume, niche ones. In the end, we’ll all use a mix and each community will find one or two mediums that work well, depending on the nature of questions, the people involved and in the end what our goals are. For examples, on SEQanswers, I don’t see a reputation really mattering that much, and any advantage to tall the rights and privileges that come with a reputation score, so a forum might prevail. On the other hand, Biostars probably works best in a Stackoverflow model.

Just don’t come up with another model. I am having a tough time keeping up.

Related articles by Zemanta

• Zillow CEO Pozits a Social Q&A Site (gigaom.com)
• The Future of Question and Answer Sites (thenextweb.com)
• Facebook Recruits Users To Test Q&A Product (businessinsider.com)
• A new Q&A website for Data-Analysis (based on StackOverFlow engine) – is waiting for you (r-statistics.com)

Anyone who reads these pages knows I am not a huge fan of the current peer-review system. IMO it is loaded in favor of incumbents, has tremendous inefficiencies and there is too much getting reviewed that doesn’t need to. However, peer review serves a purpose. As Pedro puts up in a great thread out on Friendfeed, on average peer-review adds value, even subjective opinion. Many of the commenters derive value from doing peer-review. Pedro also sees it as part of his duty as a scientist (I agree).

The problem is not with peer-review, but rather with the overall publishing ecosystem. We publish material in peer-reviewed journals that, today, can be published in a number of alternative avenues. We don’t have enough transparency and randomness in our peer review process. In the long term, to put this in terms of Glyn’s post, peer review should come in a system that deals with scarcity, as long as there is a well functioning system that deals with more abundance. To put this another way, with peer-review we should be focussing on a subset of what we publish today, while the rest goes into blogs, nature precedings, self-published articles, etc, as discussed in a previous post. The point that Glyn seems to miss is that the role of peer-review is not to just vote something up or down, but rather provide constructive, sometimes qualitative advice or suggestions, to help improve the science. I do believe that in the long run a more open peer review process might be the way to go, since we can find papers in our area of interest and provide the same suggestions via a different mechanism, but not yet. What we should rather focus on is changing the publish and reward system, which discourages scarcity, encourages “me too” and incremental publishing, and (IMO) discourages creativity.

Related articles by Zemanta

• Improving Peer Review: Let’s Provide an Ingredients List for Our Readers (scholarlykitchen.sspnet.org)
• Peer Review is No Panacea (theatlantic.com)
• Mark Midei and the failure of peer review (kevinmd.com)
• Elsevier Pilots PeerChoice in Chemical Physics Letters (prnewswire.com)

• iEvoBio
• The future of the web for collaborative science

This, coupled with some great blog posts on data analysis coming from various, non-hard science sources got me thinking.  Let’s say I am an independent researcher, and I have some cool science on hand (doesn’t have to be computational).  I can write it up and submit it to a journal.  Or I might have an opinion which I could write up and submit.  Today, we have a multitude of options about getting it out there.  What’s going to be interesting is what options we choose to get our thoughts out there.  Take Carl Boettiger’s deck on his experiments with open science.  That was posted to Precedings.  It could also have been cross-posted to slideshare or scribd.  Carl could have blogged his thoughts (expanding on the slides) and also embedded one of the slide shows.  And, nothing stops you from writing this up as an opinion paper.  The point is that we have avenues that we never had before.  What’s going to be interesting is how formal we get about all this?  Some material needs to go into Peer Review, but what happens if you blog about some of your initial results first, or maintain an open notebook.  How will it be treated?  What are the implications?

The answers are not black and white, and still in flux, but I’d encourage both scientists and publishers (and funding agencies) to rethink the current formalisms for making information available.  My hope is we’ll come up with our own set of guidelines on when, how and where without constraining creativity too much.

Related articles by Zemanta

• Papers on MapReduce algorithms (atbrox.com)
• Honu – A Large Scale Streaming Data Collection and Processing Pipeline__HadoopSummit2010 (slideshare.net)