Field of Science

What's wrong with this piece on science and journalism? Oh, let me count the ways.


An interesting conversation has been afoot addressing whether or not journalists should go to their science sources for some level of fact checking. Most journalists seem to agree that such fact-checking from their own sources should be limited and come with many caveats to protect journalistic freedom and ensure ethical practices. It's a fine line, but an experienced journalist can toe it. In this economy, many journalists must do their own fact checking.

That post over at David Kroll's Take As Directed PLoS blog and the great discussion thread that followed it led to this piece from Nature's chief online editor. The headline says, in effect, Thou Shalt not Have Your Sources Copy-Check Your Copy. The editor writes,
Imagine reading a controversial news story on climate change and later finding out that the scientist, whose findings you are reading about, had seen almost every word prior to publication.
Imagine you must because the journalists in that discussion did not describe having engaged in or supporting this practice.

A brief primer: Copy check is not fact check. When you check a fact, often all it takes is a phone call. You call the source, you summarize or read a brief section, and the source clarifies the facts. That's not copy checking. The journalists commenting on the initial blog post described a similar process. No one's sending copy wholesale for checking, that I can tell. What they are doing--and preferentially, again by phone--is asking for confirmation of factual information. I've had people do the same with me as a source.

I can see where an inexperienced journalist might flub this and cross the line into "extensive" (to quote from the Nature editor's piece) copy checking. Indeed, one of the very experienced journalists commenting on that thread mentioned having done that in her salad days. But isn't ethics a big part of teaching in J-school? It's not as though this brave new world of science journalism can't incorporate this conceptually into the curriculum and clarify the difference between fact checking with a source and copy checking.

That seems like a solution to the issue, does it not? The best source for a scientific piece covering new results is going to be the person who did that work. Does that mean you run the entire piece by the source? No. But if there is some nuance, some complexity--gee, does that ever happen in science?--that you want to make sure is accurately represented, calling that source and checking that? That would be called fact-checking. I'd do that for a business piece, a political piece, a science piece, or a fashion piece, and only three of those would, for me, carry nuance and complexity with which I might struggle (hint: Science isn't one of them).

What's not good journalistic practice, however, is telling your readers a fairy tale about how science functions. And that's exactly what happens in today's second salvo via The Guardian in which the writers, three researchers from the School of Psychology in Cardiff, make so many laughable statements about the practice of science that I almost snorfed my coffee all over my laptop. The authors appear to argue that science journalists should have their work copy-checked (not fact checked) because otherwise, the consequences of inaccuracy and overstatement will be dire. Yet, the first half of the article seems to be an argument against the need for checking.

This is where the counting begins.

1. Quote: "News stories about science are different from those about politics or business because the role of critical review has already been performed." That's the dek.
I think we all know that this excludes any stories arising from most conference proceedings, which happen to be the source of...many news stories covering science and medicine. If I had a nickel for every MSM piece I've read that scares the bejeesus out of people or sends them running for supplements that was based on conference proceedings, I'd be able to quit working.

Worth observing, were I writing a story about business or politics, I'd assume that some serious fact-checking would be required for those, as well. Science is not different from business or politics in complexity, and to me, business would involve far more esoterica requiring clarification than does science. Also, as their later argument seems to be for journalists' sending copy to scientists for checking, all of this emphasis on "already reviewed" doesn't make a lot of sense (ah, but watch where they go with it). Finally, science and business and politics, for better or worse, overlap. They are not necessarily distinct subject areas.

2. Quote: "An independent press is a cornerstone of democracy, so it is wrong for journalists to allow their sources to copy-check stories." The independence of the press arises from the press itself, and this phrase references control from external entities, such as government or a dictator. It doesn't reference how an independent press conducts itself ethically, so the first phrase in this assertion has no real link to the second. I've got no argument with the assertion that journalists shouldn't allow sources to copy-check their stories, but that's a journalistic ethics consideration, not part of the cornerstone of democracy. (ETA: As it turns out, they do have an argument with that assertion).

3. The authors then cite peer review as a "categorical" reason that science is different from, you know, stuff regular people can understand. Here's what they say:

Every research article in a reputable scientific journal has been through a process in which between two and five independent experts (normally anonymous) have made extensive comments.

These "reviewers" are looking for flaws and are often extremely critical. Many submissions are simply rejected, and if not, the authors must respond to the comments, often by making substantial changes to the article, for example by conducting more experiments, more analysis, or by presenting a different interpretation.

This process typically goes through two or more rounds, with the revised article returned to the reviewers for further comments.

A few observations. I've been on both sides of this situation, as a reviewer and as the reviewed. In addition, each week, I see reviews of research papers from many different fields and read the responses to the reviews. Add to that these at-hand examples (this and this) I've blogged of abject failures of peer review in what would probably be viewed as "reputable" journals. The world is littered with papers from reputable journals that contain statements, analyses, and experimental designs that should never have made it past peer review.

In other words, that peer-review process the authors of the Guardian piece describe above is a fairy tale. Peer reviewers are slow. They are busy. They are reluctant. They often are cursory. The usual process involves a couple of reviews that arrive, one that may be very detailed and another that may consist of a summary paragraph from a reviewer who clearly didn't actually read the entire paper. Sometimes, reviewers ask for analyses that are already described in the paper. It's not unusual to receive reviews that are completely diametrically opposed in the observations on the paper ("This is a well-written paper that will contribute to the field" and "This paper is badly written and makes no novel contribution"). I see these things weekly. There's even a Facebook page dedicated to making fun of the comments of the second peer reviewer.

It's also a fairy tale in its assumption of anonymity and purity regarding bias. Many journals ask authors to suggest reviewers for their papers. Whom do authors suggest if not colleagues in their fields whom they likely know, for better or for worse? I've had reviews in graduate school that focused more on my advisor and his reputation than on the content of the papers. In other words, their description of peer review is The Way Things Should Be. It is not, however, The Way We Live Now.

4. The authors have a pretty low opinion of science writers in general. They say that sure, there are press releases and interviews that are not peer reviewed. Ya think? In fact, those interviews are the very thing that several journalists on David Kroll's original blog post noted. They're the words that need fact checking. The writers of the Guardian article say that "wild claims are curbed by the mere fact that it would be so easy to check them against the article itself. Normally, a glance at the abstract or discussion section of the article would suffice, though we do not know how many journalists do this."

I'd hazard that a lot of journalists do this. Why wouldn't they? But it's not a way to fact check an interview or an interpretation. If it were that easy, wouldn't we just publish the research papers themselves in the MSM for people to enjoy? Regarding the statement about "wild claims" being curbed, I refer readers simply to Arsenic Life. A scientist involved in that debacle seems to have the same fantasies about science as the writers of the Guardian piece, actually referring to a "sacred boundary" that is crossed when scientists "use the media to debate questions or comments of others."

What they say here seems to muddy their argument that science is special when it comes to journalism. I mean, if all it takes is a "glance" at something, why do they refer later to "curbing" science journalists? Someone was having trouble with argument development here.

5. In keeping with their low opinion of science journalists, they say that in non-science areas, journalists are the "peer review," but that in science, the "peer review" has already occurred, so journalists are mere educators. Have I mentioned Arsenic Life yet? I think that this existence of peer review--which, as I've noted, is not consistent and not consistently a background for all science stories--is supposed to negate the role of science journalists as, you know, journalists.

6. The writers state that scientists have little to gain from exaggerated claims in the press. Reader, I give you Andrew Wakefield. On a more sober note, I give you this discussion in which Christie Wilcox argues convincingly that a "web presence" can be a key factor in a candidate's success in a faculty job search. Guess how a scientist gets a "web presence"? One way is to find yourself quoted in the MSM.

7. The writers then go on to say that avoiding inaccuracy when writing about science is a "Herculean task" relative to doing so while writing about business or politics. I disagree. If I had to write about either of the latter, the task would be more Herculean for me than cleaning out the Aegean stables.

8. They state that they believe that public trust in science and science reporting suffers more from inaccuracy than from non-independence. I assert in response that there's a reason Fox viewers use the term "MSM" with snark even as they watch Fox and read their NewsMax.

9. Then they pose what they seem to think is a dichotomous choice for journalists:
No doubt some journalists will object to our arguments because we are intruding upon their professional sphere. This is true – we are intruding. But to those who object, we would ask, what is your primary motivation? Is it simply to produce a story with an angle or is to communicate science accurately and accessibly?
Is it not possible to do both at once? I don't understand why they offer this as a set of opposing options. Like the Red Queen herself, they keep shifting back and forth, first seeming to argue that the process of science is so rigorous, so clean, so pure, that all journalists need to do is glance at things to understand them and be accurate. Then suddenly, we're faced with a reverse, an argument that science journalists must focus more on accuracy--presumably by letting their sources copy check the story--and less on angle. Again, if angle weren't an issue, wouldn't scientists just deliver research papers wholesale via the MSM? In their eyes, science journalists have no role whatsoever in the interpretation of science. They're just the "communicators."

10. Finally--and this is no laughing matter--they say: "We believe that uncurbed press freedom in science reporting...comes at an unacceptably high price." Surely I am not the only person who finds it both disturbing and cognitively dissonant that they bookend their piece with this closing statement after opening with, "An independent press is a cornerstone of democracy." After a long and winding road with many switchbacks, we arrive at their argument: Science journalists must be curbed in their walleyed misrepresentation of science, and what better way to do that than make science special as subject matter and allow journalists to cross ethical boundaries?

In the spirit of their enumeration and mine, I close with a little more numbering. These three scientists have committed three faults in this opinion piece.

1. They assume that reporting science is different from reporting on other subjects like business or politics. It is not. They all involve nuance and esoterica and all require fact checking, sometimes on the phone, even confirming quotes with sources. Science journalists are journalists, too.

2. They describe the process of peer review that exists only in Scientific Fantasy Land. Their "categorical" reason for why science is different--this process of peer review--seems to argue against their premise that science journalists should be allowed to "copy check." If the peer review process is so great, why the concern about accuracy? Indeed, their supporting assertions for their argument duck and weave until we get to what they're really trying to say, which is that science journalists, because of the existence of peer review, are not "real" journalists and instead should focus only on educating rather than interpreting.

3. They end by referring to science journalists as requiring "curbing," as though overstatement or exaggeration in journalism or headlines or tabloid copy is somehow limited only to science-related pieces. Gee, I've *never* seen an overstated political, business, fashion, or sports story, have you? Their conclusion is disturbing on many levels.

The bottom line: When fact checkers are not available, journalists must do their own fact checking. Often, this involves picking up the phone, going over the gist of a complex concept, and obtaining clarification about its accuracy. Sometimes, it involves email of a snippet or even a paraphrase that's not verbatim from the copy. No one in the original blog post at PLoS blogs was arguing for wholesale copy checks from article sources. The authors of this second Guardian piece try to make the argument that science is different and special and requires curbing of science journalists for the sake of accuracy and public education. This assertion is dangerously close to a form of fascism. I say that these authors make an excellent argument--one with which I normally tend to disagree--against letting scientists speak for themselves without a filter.

---------------------------------------------
Image credit: Roger Penguino, via Flickr.

P.S.
Over at Deep Sea News, Al Dove (a.k.a. para_sight) generated a great discussion about this whole controversy before this second piece appeared in The Guardian.

Also, for more insight into the fun and foibles of peer review, see what cog sci research fellow Jon Brock has to say about it over at Cracking the Enigma.

The ever-incisive yet always thoughtful Seth Mnookin gives a crisp overview of the history of this discussion and delivers a concise critique, promising more later today (10-12-11).

David Kroll, whose post originated the current debate, has also posted in response to this latest from The Guardian. (10-12-11)

2 comments:

  1. Wow, you, Seth, David (who I've been reading for years) AND Jon all in one crunchy package. AND on Shannon's birthday.


    More seriously, I think this is one of your strongest pieces on science writing yet.

    ReplyDelete
  2. Thanks, Liz. I guess a strong reaction drives strong writing?

    ReplyDelete

Comments for this blog are closed.

Note: Only a member of this blog may post a comment.