Skip to content

Reviewing “Meet Science: What is Peer Review?” Part One

May 8, 2011

I have awesome friends, some in the scientific community and some not.  Recently one of them posted this Boing Boing article on What is Peer Review by Maggie Koerth-Baker on my Facebook page and suggested it would be right up Tiny Science’s alley.  And it is indeed.*

I only slapped myself in the forehead with frustration a couple of times during the article, so it must not be that bad.  In fact, I’d say the Koerth-Baker article is a good summary of peer review, written from the point of view of someone outside science for others in the same situation.  Clearly scientists were interviewed for the piece, and clearly none of those scientists was me, so I’m taking the opportunity to respond here on the blog.  If you want, you can read the original article and then come back and join the discussion.

Part One, in which the topic “what is a scientific journal article” is discussed.  Quotes from the article and comments section are pulled out in italics:

“Journal articles are like book reports. . . ”

Sigh.  When the meat of the article begins with these words, it’s almost enough to stop a girl from reading the whole thing.  Book reports?  Seriously?  Comparing the papers of Einstein’s miracle year and the article detailing the discovery of the structure of DNA to something done by fourth graders after skimming the Cliff’s Notes of Across Five Aprils??  Scientific discoveries written up in formal reports have saved millions of lives and changed the way we see the world and ourselves.  Let’s just forget that those words even happened and move on to the rest of the first sentence.

“. . . usually written to document the methodology and results of a single scientific experiment, or to provide evidence supporting a single theory.”

Oh how I wish that a single successful experiment was enough to get me a journal article!  I would have so many publications I would not know what to do with myself.#  The second part of the sentence is actually closer to reality.  An article provides evidence supporting a single hypothesis.  Hypothesis.  Not theory.  Please, for the love of all that is good and right in the world, STOP MIXING UP THEORY AND HYPOTHESIS.  The scientific community is all out of bubble gum^ on this topic. 

Journal articles are the story of a scientific idea.  A good scientific report starts with a description of the problem the authors are trying to address, put into the context of work that they (and other groups in their field) have already published.  Then it describes the series of experiments the authors did and the evidence they gathered from those experiments.  In the end, the authors try to explain how they interpret all the data related to the idea under examination (the results of their experiments and analysis as well as other evidence and analysis published by other groups) and why they think their interpretation has merit as a solution to the problem they set for themselves.  Peer review looks at all aspects of the report, including:

  • The validity of the original question.  Has this question been asked before? Does this article provide a unique perspective on the problem?
  • The reasonableness and reliability of the experiments done.  Do the authors provide proper controls? Do the experiments address the problem?  Are there other experiments that should have been done to address the problem from a different angle?
  • The authors’ interpretation of their data.  Are the results of the experiments conclusive, or are there other interpretations that the authors do not address?  How should those interpretations be handled?  Should more experiments be done, or can some wording be put in that discusses the limitations of the current work?
  • The authors presentation of the current knowledge in their field and how their work fits into that context.

If the reviewers think the article failed to achieve a certain (unwritten) standard in each of these categories, they can either reject the article for publication outright, or suggest ways that the article can be improved before publication.  Addressing reviewer comments in the first three categories can take a couple of months’ worth of experiments and data analysis, but usually makes the article stronger.  Comments in the last category are mostly an opportunity for reviewers to complain that their work was not cited, or that the authors are making their work out to be more important to the field than it really is.  A brief snort and a change in the wording of the article is usually all that is necessary here.  Hey, even the best scientists can get petty.  We’re humans, after all.  More on that later.

“No single journal article is meant to be the definitive last word on anything.”

Well thank goodness someone finally said it out loud.  Although if science journalists know this, why do they continue to do reports on single articles where the only indication that the article being discussed is not the be all and end all on a topic is the phrase “more research needs to be done.”?  If “peer reviewed journal article” is meaningless to laypeople, what does that trite little phrase mean?

I’ll tell you what I think it says to most laypeople.  If they agree with and like the conclusions presented in the report about the article, then “more research” means the conclusions will be supported by other people and “science” will have discovered a “truth.”  If they don’t agree with or don’t like what the report says, “more research” means, “don’t listen to a word we just said about this discovery, because when we do more research, it will just turn out to be wrong and everyone will go back to not worrying about it.”  Honestly, I think that is what “more research” means to most laypeople.  Laypeople, please, please write in and tell me that I’m cynical and wrong.  Tell me what you are thinking as you hear that phrase.  I genuinely want to know.

Now that the first part of the article detailing what a scientific journal article is has been given the Tiny Science treatment, I’m going to take a break and start a new post.  In Part Two, we’ll discuss the rest of the article, which is a nice summary of peer review with emphasis on who reviewers are, what they are supposed to do (vs. what they really do) and the pros and cons of the current peer review system.

If you’d like to discuss Part One, let’s dialogue in the comments section.  Remember that I’m a biologist, so my take on all this is from the biomedical sciences perspective.  I would love for people from other disciplines to weigh in with their take on the peer review process, and for laypeople to get into the discussion as well.

———-Notes———

Yes, I footnote certain blog articles.  It lets me add snark without interrupting the flow of the article.

* I have written an article on peer review, and a couple of articles about science journalists’ take on scientific culture (see them here, here, a funny take by someone else linked to here, and my personal favorite here).

# As it is I’m only listed as an author on ten papers.  This averages out to one a year for the twelve years I’ve considered science my full-time job.

^ B-grade horror flicks have awesome catchphrases.  Haven’t seen the John Carpenter movie “They Live”?  You’re not missing anything, except for this one line, which through the magic of YouTube you can see without waiting for Halloween on AMC.

Advertisements
7 Comments
  1. April permalink

    Sometimes I forget there are people who didn’t learn about journal article writing in school.

    Seriously “more research needs to be done” is what I used to write at the end of my freshman year lab reports because I didn’t have time to read all the assigned material to adequately make a conclusion as to why my experiment failed. (Though I quickly came to the conclusion in my E&M lab that if Dr. Hart would stop wearing sweaters and looming over the experiments we would have had better results.)

    From what I’ve gathered in my time in Physics and Geology the processes for peer reviewing articles are basically the same as for you squishies. I wanted to add something though, someone who is asked to peer review an article (in the fall and get hurt sciences anyway) is usually considered an expert in the same field as the article or if the article is in a newish field with few experts the reviewers are experts in similar or related fields. Mostly this is so the reviewers can call shenanigans on the articles if they’re just making stuff up.

    Also the person asked to review a paper can refuse for any number of reasons including “I had this person as a grad student and he was a moron, I can’t imagine he’s gotten any better.”

    I’ve heard of people refusing to review articles on subjects hoping to get their own similar research published first, and articles that have been challenged by reviewers because someone thought their work had been stolen or pirated.
    What can I say? In many universities the rule is “publish or perish” and journals don’t want to run two articles on the same topic in back to back issues.

    • I think the key point is that more research ALWAYS needs to be done, one article is very rarely the seminal work in a field. I would hope that most scientists past their Bachelor’s know this, and I think that a lot of science journalists know this, they just don’t say it because it doesn’t fit into a soundbite. And that’s what the commentary is really about, the journalists that report science to the public, not so much the scientists themselves.

      I propose that instead of saying “more research needs to be done” it would be great if science journalists could give a little context for the article and say where it gets us in the field, and what sort of research needs to be done, instead of just “more.”

      I also agree with your points on the dirty underbelly of peer review, and the hijinks that go on behind the scenes. It’s more political than it should be sometimes. More on that in part two. . .

  2. April permalink

    Yes I failed coma class; so sue me.

  3. April permalink

    comma

    • Good that you failed comma class and not coma class. Failing a class about being in a coma. . . I don’t know if that’s a good thing or a bad thing.

  4. April permalink

    Apollo (the laptop) doesn’t seem to like when I type double letters.

Trackbacks & Pingbacks

  1. Reviewing “Meet Science: What is Peer Review?” Part Two « Tiny Science

Comments are closed.

%d bloggers like this: