For a long time now, I've (LinkedIn, Google) been thinking that there are problems with the academic peer review process. I'm not the only one and I'm certain that you've been thinking the same thing whether you're a professional academic, graduate student, science journalist, working in industry, or a knowledge enthusiast. A number of recent, very public discussions (see The Economist, Columbia University, and Science for examples) have taken place that have been critical of modern academe for the way it handles the production, evaluation, and dissemination of the knowledge it's meant to be producing—something the institution often does at tax-payer's expense, which suggests that it should be A) productive, B) trustworthy, and C) accountable. However, these are not characteristics that many people would attribute to the current peer review system without some reservations.

While the ills of peer-review are many, we can solve a lot these problems by adding a simple layer to the knowledge dissemination process. It won’t cost much, it won’t take long, and it will make a BIG difference.

Imagine this: you sit down with your coffee in the morning to read through some headlines and find a couple of reports on published research. You might be a professional academic or an end-user in industry. You’ve managed to access the research paper in question and you give it a quick read. At that moment, you have no one to bounce the ideas off of. You might not even fully understand every aspect of the paper (even if you won’t admit that your grasp of hierarchical structural equation models isn’t what it should be), but the results bear on something of interest to you. Perhaps you fully understand the paper, but you’re limited by time, or your research interests and experience limit your perspective (who’s don’t?); so, how can you be sure that you believe the results and conclusions of the paper? I’m certain this has happened to everyone. Regardless of how well read you are, or how expert your knowledge is, you are but one individual trying to consume information that may have taken multiple people many years to produce. There is no way you’re going to catch everything. But then you remember there’s a fantastic resource for getting a quick overview of how other people, experts and enthusiasts alike, view the paper. You get on the Internet and head over to the ‘rotten tomatoes’ of academe.

The goal of the project is simple: create a website that will serve as the first port-of-call for evaluating published academic research. The end-user, that’s you, will be able to register on the website, submit citations for review, and review any citations that have been posted. A simple, but comprehensive, set of review categories will be made available using a straightforward percentage rating system. The categories will include things like “quantitative rigour”, “numerical accuracy”, “reproducibility”, “internal support for conclusions”, and so on. Categories will be tailored to different fields and the community of registered users will have a say about what the categories should be. A minimal, but indispensable, set of categories will be created for each category of paper (Social Science General, Psychology, Anthropology, Biology, Environmental Science, etc.). Registered users will be able to write extensive comments in their reviews and be as general or specific as they like. The review ratings will be collated and presented alongside the citation, abstract, and access information for each article. The reviews can be sorted and searched on the basis of ratings, title, publication date, etc.

The standard peer-review process is still going to be the gatekeeper to academic publishing, but our new web facility will be the most important new development in the dissemination of science since peer-review was established. We will all use the reviews to help us evaluate the published research and see how a broader community of minds is responding to the work—that of others and our own. The public will benefit from having easy access to the reviews so that they can better assess the research that appears in public forums. The scientific process, as a whole, will benefit from this kind of democratization. No longer will the fact that something has appeared in an academic journal be the only rapid way to assess its validity, its impact, or expert consensus regarding the work.

If you’re already convinced that you want to be a part of the academic revolution, pledge some money! Here’s what you’re paying for:

Website development (WordPress; ~600 CAD), website hosting on Bluehost (~50 CAD for a year), promotions (via Facebook, LinkedIn, and Academia.edu; ~200 CAD) to help get the word out, and transaction costs associated with FundRazr (~100 CAD if we only just meet our goals). The site will be up and running within 8 weeks of this campaign reaching its funding goal.

If you want to know more, read on.

Some of the initial, short term, plans for the structure of the website are as follows:

•        Users who want to post citations for review or review the existing citations will have to register. Registration will involve two categories: verified and unverified. The purpose of verification is that credentialism is an important part of society at large and science in particular, as much as we might disagree with it sometimes. Registered users will be able to create social media style profile pages and if someone claims to hold two PhDs in Quantum Mechanics, then the community has a right to know if those credentials can be verified. Aside from the credentials, it also says something if you’re willing to have your identity associated with a comment. We will recognize that anonymity has its place, but verifying IDs and credentials increases the utility of the site for visitors who just want to glance at the review stats instead of reading every review in detail to evaluate the counter arguments on their own grounds. We will also provide Facebook, Academia.edu, and LinkedIn support to bring the website seamlessly into your online networks.

•        Review stats will be divided into verified, unverified, and total summaries so that you can tell at a glance what the two categories of registered users think about a paper.

•        Any registered user can submit citations. The person who submits a citation for review will not be given special placement or prominence for their review. Their review will be posted below the entry and collated with the others.

•        Each entry to be reviewed will have its full citation information published on the website (title, authors, affiliations, abstract, and access information including links to relevant websites).

•        Each reviewer will also be asked whether they have replicated the work in the paper in whole or in part and to provide details. Sometimes, this might involve only double-checking the original author’s algebra and other times it might involve a full replication. If a reviewer declares that they have replicated the study, full evidence will, of course, be required and explanations should appear in the reviewer’s comments section.

•        The website will contain a forum for fuller discussions about papers.

•        I have also considered creating a section for submitting draft papers for review. In this case, the full paper will be made available assuming there are no copyright issues.

•        On the main page, there will be options for sorting the reviews according to type, popularity, posting date, and rating statistics. There will also be ‘top ten’ lists for each field.

•        For now, moderation of the site and comments (for language and appropriateness only) and general administration will be handled on a volunteer basis.

•        The site will be highly responsive, constructed with WordPress/Drupal, aesthetically pleasing, minimalistic, devoid of ads and clutter, and simple to use.