Prove it! The burden of proof in science vs pseudoscience disputes

Time to look back at one of my technical papers, this one published in 2013 with my friend and collaborator Maarten Boudry in the journal Philosophia, and entitled “Prove it! The burden of proof in science vs Pseudoscience disputes.” (As with all my technical papers, they can be downloaded from my DropBox, by going to this link.)

The starting point for the paper is that the concept of burden of proof is used in a wide range of discourses, from philosophy to law, science, skepticism, and even in everyday reasoning. Maarten and I, then, provide an analysis of the proper deployment of burden of proof, focusing in particular on skeptical discussions of pseudoscience and the paranormal, where burden of proof assignments are most poignant and relatively clear-cut. We argue that burden of proof is often misapplied or used as a mere rhetorical gambit, with little appreciation of the underlying principles. The paper elaborates on an important distinction between evidential and prudential varieties of burdens of proof, which is cashed out in terms of Bayesian probabilities and error management theory. Finally, we explore the relationship between burden of proof and several (alleged) informal logical fallacies. This allows us to get a firmer grip on the concept and its applications in different domains, and also to clear up some confusions with regard to when exactly some fallacies (ad hominem, ad ignorantiam, and petitio principii) may or may not occur.

To begin with, an important distinction needs to be made between prudential and evidential burden of proof (BoP). The prudential BoP is applicable when there are cost asymmetries in arriving at two judgments about whatever matter is under dispute, whereas the evidential burden of proof applies when there are no such cost asymmetries involved.

Consider, for instance, the question of the safety of food additives. If approached as a straightforward scientific question, then the relevant concept is that of evidential BoP: there is no “cost” associated with arriving at the right judgment, other than the symmetric cost in getting a chunk of reality wrong. But if we approach the issue of food additives from the standpoint of its potential consequences for public health, there is a differential cost in getting the wrong answer, so the idea of prudential BoP seems more appropriate.

The (controversial) precautionary principle, which is an application of the prudential burden of proof, states that — if a certain action or policy is suspected to be harmful — the burden falls on those who believe that a new policy or course of action is not harmful. The status quo is perceived as less costly than a potentially dangerous new policy or course of action. In more general terms, the prudential BoP can be applied in situations where the cost of a false positive is significantly different (greater or smaller) from the cost of a false negative.

Examples of prudential BoP where the cost associated with a false negative outweighs that of a false positive include smoke detection alarms, environmental hazards, cancer screening, etc. An example of the opposite case, where false positives are perceived as more costly, include the presumption of innocence in a court of law. This principle in American criminal law clearly skews things in favor of the defendant, but this is done because the risk of a false positive (convicting an innocent) is treated as much less acceptable than the risk of a false negative (exonerating a guilty party).

Of course, cases of prudential BoP always involve an evidential dimension as well, while the opposite is not the case. In prudential BoP, cost asymmetries have to be taken into account in addition to prior probabilities. For example, in discussions about cancer and cell phones, the initial plausibility of low-energy electromagnetic radiation being carcinogenic has to be taken into account in addition to cost asymmetries. If prior probabilities are ignored, the precautionary principle is misused and can have paralyzing effects on public policy. Conversely, one cannot just invoke a Bayesian perspective (as useful as it is) to settle issues where cost asymmetries are involved, since even when competing claims have equal priors, a prudential approach (but not an evidential one) could easily tip the balance in favor of one claim over the other.

There are a number of important discussions in science, pseudoscience, and even in straightforward philosophical argumentation, that can reasonably be approached either from an evidential or from a prudential perspective, depending on the interest of the parties involved. For instance, the force of the philosophical argument behind Pascal’s wager is supposed to be that the risk of a false negative (you don’t believe there is a god, but it turns out there is one) is much higher than that of a false positive (because of the threat of eternal damnation in Hell). By contrast, to take another philosophical example dealing with the import of paranormal or supernatural hypotheses: the risk (in terms of practical consequences) of falsely accepting the existence of Bertrand Russell’s tea pot orbiting the sun (false positive) seems to be the same as the risk of rejecting the tea pot when there really is one (false negative).

Maarten and I then discuss a number of sources of subjectivity in the judgment of were the burden of proof lies, as well as a distinction between “global” and “local” burden of proof, where the global BoP is fixed throughout a discussion, because it is related to what a discussant ultimately wishes to establish (or her opponents wishes to deny). Within that broad goal, however, a number of local burdens of proof may arise, which shift during the debate itself, as they pertain to smaller pieces of the overall puzzle.

We move on to consider how BoP should be assigned. In 1970, Brown characterized the request for meeting the burden by a given side in a debate as amounting to the claim that, prima facie, that side’s position is more initially plausible than the alternative(s). Brown’s framework does not involve the costs associated with different judgments, and can thus be seen as a characterization of evidential BoP. A major exponent of modern skepticism, Michael Shermer, describes the principle of (evidential) BoP as follows: “The person making the extraordinary claim has the burden of proving to the experts and to the community at large that his or her belief has more validity than the one almost everyone else accepts.” Psychologist Terence Hines, in another compendium on pseudoscience, agrees that the burden should fall on the claimant of the extraordinary, because “it is often impossible to disprove even a clearly ridiculous claim,” such as that Santa Claus exists.

We discuss a formal analysis of these ideas, carried out by Larry Laudan (details in the paper), but warn that such an analysis should not be cause for too much complacency on the part of the skeptic of pseudoscience, since it doesn’t license an automatic rejection of any claim of the paranormal or extranormal, except when the prior probability of the paranormal hypothesis is exactly zero (e.g., when it is logically incoherent). The reason why BoP rests on the believers is also often misconstrued in the skeptical community. The evidential BoP is not on “whoever makes the positive claim.”

First, it is very easy to turn any positive claim into a negative one, and vice versa, by simple application of basic logical rules. In general, affirming P is exactly the same as denying ~P. Any existential claim can be translated into a negative universal, and vice versa. Resorting to such moves would merely amount to sophistic word play rather than a substantive consideration of epistemic burden.

Second, there are cases in which the BoP rests on those who are putting forth what may most plausibly be construed as the “negative” claim, in the sense of denying the material existence of some X. For example, the burden of proof is no longer on historians to provide evidence of Zyklon B use in the nazi concentration camps, although, apart from logical sophistries, they are the ones making a “positive” claim. In this case, then, the BoP rests on those making the “negative” claim.

In most discussions of pseudoscience and the paranormal, admittedly, the believers in pseudoscientific notions are making positive claims, in the sense of affirming the existence of entities (spaceships, psi force, qi energy lines, auras) that are rejected by modern science, but this — per se — is not the reason why the BoP rests on them. Evidential BoP assignment always reflects substantial background knowledge and prior probabilities, and these assumptions of plausibility, we argue, should be based on the expert consensus on the matter.

Maarten and I go on explore the role of Occam’s razor in this debate and then tackle what we call the technical burden. Believers of the paranormal and supernatural have often tried to turn the tables on skeptics, finding various ways to shift the BoP back to the latter. In particular, rhetorical moves of the type “you can’t prove it wrong” are unfair requests that fail to appreciate the proper BoP procedure. In some cases, such requests can be straightforwardly fulfilled (e.g., it is very easy to prove that the co-authors of this paper, at this very moment, have far less than $1 M dollar in their pockets), but even then, the skeptic is doing the accuser a favor in taking on a BoP that does not really fall on him (we are under no obligation to empty our pockets after each such gratuitous insinuation).

Similarly, if ufologists claim that some crop circle was left by a space ship, the BoP is firmly on their side to come up with extraordinary evidence. If the skeptic chooses to take on their sophistic challenge to “prove that there was no spaceship” by way of providing direct or circumstantial evidence that that particular crop circle was in fact a human hoax, they are indulging the believers by taking on a BoP that, rationally speaking, does not pertain to them at all.

For most actual para/extranormal claims, however, the space of possibilities cannot be exhausted in a finite (and suitably short) time. For instance, to arrive at proof that there are no alien spaceships visiting earth — at any moment, not just in the case of a specific alleged incident — would require a type of temporally protracted exhaustive monitoring of the entire planet’s surface, something that it is so far beyond current technological possibility that the request can easily be dismissed as a simple debating trick.

This, however, leaves the skeptic with a dilemma. Although it may sometimes be rhetorically persuasive for her to take on a BoP that, strictly speaking, does not fall on her (for example, providing a natural explanation of a given UFO sighting), this may be perceived as an implicit acknowledgement that skeptics do carry the negative BoP for every single anomaly that believers come up with. The result is a mug’s game for skeptics: all believers have to do is throw around challenges for the skeptic, who will surely not be able to answer every single one of them. To refer again to the ufological literature, even ardent skeptics do admit that a small percentage (at most 10%, and likely significantly less than that) of alleged UFOs cannot be turned into IFOs (Identified Flying Objects), even after direct investigation of the available evidence.

There are at least three replies the skeptic has available here. To begin with, investigative resources are limited, especially when it comes to likely pseudoscientific claims, so it should not be surprising that on a certain number of occasions the researcher simply does not have sufficient means to carry out a positive identification of the allegedly unexplained phenomenon.

Second, even in the case of genuinely scientific questions one has to contend with limited epistemic access to the relevant phenomena, access that can be affected by the lack of sufficient empirical traces or by the intrinsic epistemic limits of human reason. Think of the long — and so far still largely unsuccessful — quest for an explanation for the origin of life, for instance.

Third, as Thomas Kuhn reminded us, even successful “normal” science constantly has to deal with a number of unsolved “puzzles,” and it is only when the puzzles become numerous and widespread that they genuinely begin to threaten the reigning paradigm, forcing scientists to seek alternative theoretical frameworks. Even if skeptics cannot provide a complete explanation for every single anomaly, what they often can do is to offer promissory notes for explanations, speculating about potential natural interpretations. Given that the BoP really falls on believers to come up with convincing evidence, this is all that can be expected from skeptics under these circumstances.

Intelligent Design proponents and assorted creationists, for instance, have often pointed to alleged instances of “irreducible complexity” in the living world: biological systems that are so intricate that they could not possibly have evolved. In dealing with such challenges, evolutionary biologists can suggest possible evolutionary pathways leading to a given complex biological structure. When they have done so, there is an extra BoP on ID advocates to rule out all of the proposed natural explanations. Contrary to what believers think, the BoP is not on skeptics to demonstrate which one of the natural explanations is the correct one. Given the overwhelming evidence for the power of natural selection to produce adaptive complexity, and the difficulty of garnering information about a distant evolutionary past, this kind of informed speculation is all that is needed to put ID arguments to rest (of course, evidence of specific mutations and selection processes further strengthens the case for evolution, but its fate no longer depends on it). The amount of anomalies (in casu, evolutionary puzzles) has simply not come even close to the Kuhnian threshold for a paradigm shift, though of course this says nothing about whether it might do so in the future.

At this point the paper changes direction somewhat, and Maarten and I provide a discussion of so-called informal logical fallacies. I strongly suggest interested people to check the paper for the details, but we basically argue that too often skeptics (and now, increasingly, believers) throw out the “you committed logical fallacy X” as if that were the end of all discussion. Sometimes an informal fallacy is not a fallacy at all, but actually a good heuristic, or a relevant piece of information.

Let’s say, for instance, that during court proceedings a lawyer for the defense points out that a hostile witness has a history of being unreliable and of lying, or perhaps has something to gain if the accused is convicted. The prosecution can’t just shout “ad hominem!” and be done with it, since information about the character and/or personal interests of the witness are, in fact, germane to the case, even though of course they don’t prove that the witness is lying on this particular occasion.

In conclusion, the word “skepticism” has, of course, a long and venerable history in philosophy. When it comes to disputes about allegedly pseudoscientific notions, though, the term may refer to one of two distinct attitudes: one corresponds to someone who knows that the para- or extra-normal claim is wrong and is out to prove it. Although this may in fact be the case in many actual instances, such a figure is not at all intellectually interesting. The second meaning is the Humean sense in which “a wise man proportions his belief to the evidence.” If we are to be honest Humean skeptics, though, we need to set the bar for evidence of extraordinary claims at the right level, not as low as a gullible believer would wish it, but not as high as for the BoP to be impossible to meet.

Modern skeptics are fond of quoting Carl Sagan’s rendition of the Humean dictum mentioned above: “Extraordinary claims require extraordinary evidence.” This is fine as far as it goes, but we clearly need criteria to credibly establish when a claim is indeed “extraordinary,” and what would count as commensurate evidence. Hume’s own famous argument against miracles is sometimes (uncharitably, we think) interpreted as amounting to a statement of the impossibility, not just very low likelihood, of miracles, and people who believe in ufological or paranormal phenomena echo that sentiment when they claim that skeptics will never be satisfied no matter how compelling the evidence is going to be.

However, Hume’s approach in Of Miracles can be reasonably reformulated in Bayesian terms, with the priors — and consequently the BoP — being set by the accepted background conditions pertinent to the dispute at hand. Seen from this perspective, all we need to avoid are the extremes of setting our priors to 0 (complete skepticism) or to 1 (complete belief), since no amount of data can possibly move us away from those limit cases. Indeed, there are some instances in the skeptical literature on pseudoscience where priors have significantly moved over time. For instance, while acupuncture is still criticized in terms of both the underlying theory and the exaggerated claims of its supporters, there may now be sufficient evidence of its limited efficacy that a skeptic needs to reconsider outright rejection. This is even more so for a variety of transcendental meditation techniques, where again one may reasonably reject the underlying metaphysics while agreeing that qua techniques they do work for a range of claimed effects.

If anything, it is harder to find prominent exponents of para- or extra-normal beliefs that have changed their mind in the face of skeptical arguments (though even those can be found, if one digs deep enough). Which brings us to the last point in this paper (which I haven’t discussed above): discussions of BoP in the context of science vs pseudoscience disputes are, of course, a type of Wittgenstenian language game that presupposes a minimum commonality of standards. People cannot agree on how to fairly allocate BoP unless they find themselves at the least in the same ballpark when it comes to the type of background knowledge that constraints the priors pertinent to the dispute at hand. And that is precisely the most common obstacle in debates between skeptics and believers: the former too often simply reject out of hand even the possibility of an anomalous phenomenon turning out to be real, while the latter are equally quick to label the entire scientific enterprise as “too reductionist” or narrow minded to be able to come to terms with novel phenomena. This sort of impasse depends on a widespread lack of appreciation for the sort of epistemic issues Maarten and I have described in this paper, but it also boils down at least in part to individual psychological attitudes, whereof a philosopher is better served not to speak.