Reviewing “Meet Science: What is Peer Review?” Part One
I have awesome friends, some in the scientific community and some not. Recently one of them posted this Boing Boing article on What is Peer Review by Maggie Koerth-Baker on my Facebook page and suggested it would be right up Tiny Science’s alley. And it is indeed.*
I only slapped myself in the forehead with frustration a couple of times during the article, so it must not be that bad. In fact, I’d say the Koerth-Baker article is a good summary of peer review, written from the point of view of someone outside science for others in the same situation. Clearly scientists were interviewed for the piece, and clearly none of those scientists was me, so I’m taking the opportunity to respond here on the blog. If you want, you can read the original article and then come back and join the discussion.
Part One, in which the topic “what is a scientific journal article” is discussed. Quotes from the article and comments section are pulled out in italics:
“Journal articles are like book reports. . . ”
Sigh. When the meat of the article begins with these words, it’s almost enough to stop a girl from reading the whole thing. Book reports? Seriously? Comparing the papers of Einstein’s miracle year and the article detailing the discovery of the structure of DNA to something done by fourth graders after skimming the Cliff’s Notes of Across Five Aprils?? Scientific discoveries written up in formal reports have saved millions of lives and changed the way we see the world and ourselves. Let’s just forget that those words even happened and move on to the rest of the first sentence.
“. . . usually written to document the methodology and results of a single scientific experiment, or to provide evidence supporting a single theory.”
Oh how I wish that a single successful experiment was enough to get me a journal article! I would have so many publications I would not know what to do with myself.# The second part of the sentence is actually closer to reality. An article provides evidence supporting a single hypothesis. Hypothesis. Not theory. Please, for the love of all that is good and right in the world, STOP MIXING UP THEORY AND HYPOTHESIS. The scientific community is all out of bubble gum^ on this topic.
Journal articles are the story of a scientific idea. A good scientific report starts with a description of the problem the authors are trying to address, put into the context of work that they (and other groups in their field) have already published. Then it describes the series of experiments the authors did and the evidence they gathered from those experiments. In the end, the authors try to explain how they interpret all the data related to the idea under examination (the results of their experiments and analysis as well as other evidence and analysis published by other groups) and why they think their interpretation has merit as a solution to the problem they set for themselves. Peer review looks at all aspects of the report, including:
- The validity of the original question. Has this question been asked before? Does this article provide a unique perspective on the problem?
- The reasonableness and reliability of the experiments done. Do the authors provide proper controls? Do the experiments address the problem? Are there other experiments that should have been done to address the problem from a different angle?
- The authors’ interpretation of their data. Are the results of the experiments conclusive, or are there other interpretations that the authors do not address? How should those interpretations be handled? Should more experiments be done, or can some wording be put in that discusses the limitations of the current work?
- The authors presentation of the current knowledge in their field and how their work fits into that context.
If the reviewers think the article failed to achieve a certain (unwritten) standard in each of these categories, they can either reject the article for publication outright, or suggest ways that the article can be improved before publication. Addressing reviewer comments in the first three categories can take a couple of months’ worth of experiments and data analysis, but usually makes the article stronger. Comments in the last category are mostly an opportunity for reviewers to complain that their work was not cited, or that the authors are making their work out to be more important to the field than it really is. A brief snort and a change in the wording of the article is usually all that is necessary here. Hey, even the best scientists can get petty. We’re humans, after all. More on that later.
“No single journal article is meant to be the definitive last word on anything.”
Well thank goodness someone finally said it out loud. Although if science journalists know this, why do they continue to do reports on single articles where the only indication that the article being discussed is not the be all and end all on a topic is the phrase “more research needs to be done.”? If “peer reviewed journal article” is meaningless to laypeople, what does that trite little phrase mean?
I’ll tell you what I think it says to most laypeople. If they agree with and like the conclusions presented in the report about the article, then “more research” means the conclusions will be supported by other people and “science” will have discovered a “truth.” If they don’t agree with or don’t like what the report says, “more research” means, “don’t listen to a word we just said about this discovery, because when we do more research, it will just turn out to be wrong and everyone will go back to not worrying about it.” Honestly, I think that is what “more research” means to most laypeople. Laypeople, please, please write in and tell me that I’m cynical and wrong. Tell me what you are thinking as you hear that phrase. I genuinely want to know.
Now that the first part of the article detailing what a scientific journal article is has been given the Tiny Science treatment, I’m going to take a break and start a new post. In Part Two, we’ll discuss the rest of the article, which is a nice summary of peer review with emphasis on who reviewers are, what they are supposed to do (vs. what they really do) and the pros and cons of the current peer review system.
If you’d like to discuss Part One, let’s dialogue in the comments section. Remember that I’m a biologist, so my take on all this is from the biomedical sciences perspective. I would love for people from other disciplines to weigh in with their take on the peer review process, and for laypeople to get into the discussion as well.
Yes, I footnote certain blog articles. It lets me add snark without interrupting the flow of the article.
* I have written an article on peer review, and a couple of articles about science journalists’ take on scientific culture (see them here, here, a funny take by someone else linked to here, and my personal favorite here).
# As it is I’m only listed as an author on ten papers. This averages out to one a year for the twelve years I’ve considered science my full-time job.
^ B-grade horror flicks have awesome catchphrases. Haven’t seen the John Carpenter movie “They Live”? You’re not missing anything, except for this one line, which through the magic of YouTube you can see without waiting for Halloween on AMC.