I quote from: Jacob Cohen, The Earth Is Round (p < .05), American Psychologist 49, no. 12 (1994), 997-1003:

One problem arises from a misapplication of deductive syllogistic reasoning. Falk and Greenbaum (in press) called this the “illusion of probabilistic proof by contradiction” or the “illusion of attaining improbability.” [...] It is the widespread belief that the level of significance at which Ho is rejected, say .05, is the probability that it is correct or, at the very least, that it is of low probability.

The following is almost but not quite the reasoning of null hypothesis rejection:

If the null hypothesis is correct, then this datum (D) can not occur.It has, however, occurred.Therefore, the null hypothesis is false.

If this were the reasoning of Ho testing, then it would be formally correct. It would be what Aristotle called the

*modus tollen**s*, denying the antecedent by denying theconsequent. But this is not the reasoning of NHST. Instead, it makes this reasoning probabilistic, as follows:

If the null hypothesis is correct, then these data are highly unlikely.These data have occurred.Therefore, the null hypothesis is highly unlikely.

By making it probabilistic, it becomes invalid. Why? Well, consider this: The following syllogism is sensible and also the formally correct

*modus tollens*:If a person is a Martian, then he is not a member of Congress.This person is a member of Congress.Therefore, he is not a Martian.

Sounds reasonable, no? This next syllogism is not sensible because the major premise is wrong, but the reasoning is as before and still a formally correct

*modus**tollens*:

If a person is an American, then he is not a member of Congress.(WRONG!)This person is a member of Congress.Therefore, he is not an American.

If the major premise is made sensible by making it probabilistic, not absolute, the syllogism becomes formally incorrect and leads to a conclusion that is not sensible:

If a person is an American, then he is probably not a memberof Congress. (TRUE, RIGHT?)This person is a member of Congress.Therefore, he is probably not an American. (Pollard &Richardson. 1987)

This is formally exactly the same as

If Ho is true, then this result (statistical significance) wouldprobably not occur.This result has occurred.Then Ho is probably not true and therefore formally invalid.

This formulation appears at least implicitly in article after article in psychological journals and explicitly in some statistics textbooks—”the illusion of attaining improbability.”

(Quoted from: Jacob Cohen, The Earth Is Round (p < .05), American Psychologist 49, no. 12 (1994), 997-1003.)

I remember liking that paper when I first read it (in fact, you probably gave it to me).

Surely, though, in a hypothesis test the situation is designed so that H_0 and H_1 are contradictory and H_1 leads to the obtained data being (relatively) likely. The americans/congress example fails this: H_1 here is “the person is not an American”, which actually reduces the probability of the observed outcome (member of congress).

This is not to assert that the same logical error isn’t widely made in the literature, of course.

By:

Dr Rickon January 2, 2010at 10:10 am

So what is the correct conclusion in the probabilistic American/Congress example?

By:

Traveleron January 2, 2010at 5:56 pm

The paper quoted contains an explanation. It is less trivial than I would like to have.

By:

Alexandre Borovikon January 3, 2010at 6:16 am