February 05, 2006

Peer Review

from - RSA

In another forum I've been arguing with a Creationist about evolution, and I came to the realization that not everyone understands the idea of peer review. This is scary in that some of the people who are running the nation's scientific and regulatory agencies also seem to have no grasp of the idea.

From the LA Times:

Cal/EPA's air pollution epidemiology chief, Bart Ostro, charged during the teleconference that the EPA had incorporated "last-minute opinions and edits" by the White House Office of Management and Budget that "circumvented the entire peer review process."

In principle, peer review is easy to understand. Scientific results generated by Researcher X are not taken on faith by the scientific community, but rather are reviewed by X's peers before they make their way into the literature. When you read a peer-reviewed scientific article, you know that it's been vetted by scientists who have looked at the claims in the article and found them sound.

One might think that this is an authoritarian, faith-based approach to science, that the high priests of science must give their imprimatur to some piece of work before it can appear, but this is not the case. The keys are transparency and appropriate incentives. For an analogy, suppose that you and your roommate are going out for the evening. He's been out earlier, so you ask him, "Is it raining outside?" He says, "No." Is it a matter of faith for you to believe your roommate? Not at all: he's been trustworthy in the past; he has a stake in telling you the truth (he'll get wet too, if he dresses so as to mislead you); his answer can be checked directly, if you look out the window.

The same applies to scientific research. Reviewers have a track record in their area, or they wouldn't be viewed as competent for reviewing new work. Because reviewers tend to be chosen from the same area as the work they're reviewing, they have a stake knowing whether they'll be able to rely on the results or not. Reviewers usually want enough information to be able to reproduce the results they're reviewing. All this means that, if I trust the peer review process, I can trust the results that come out of it.

Now we come to the Bush administration, which has not been a friend to science in general. They seem to take the shallowest view of scientific research: It's just a bunch of opinions by people who think they're smarter than we are, and what's more, they're people who tend not to vote Republican, so screw 'em. Is this a fair characterization? I think it's consistent with the Bushies' preference for politics over policy, appearance over substance, and happy talk over hard truths. I should probably spend some time documenting this, but others are already doing a fine job, including Chris Mooney at the Intersection and Tim F at Balloon Juice.


Posted by RSA at February 5, 2006 11:00 AM
Comments for this entry are closed. Please leave your notes on a more recent comment thread.