Posted by: Alexandre Borovik | January 2, 2010

The fallacy of null hypothesis rejection

I quote from: Jacob Cohen, The Earth Is Round (p < .05), American Psychologist 49, no. 12 (1994), 997-1003:
One problem arises from a misapplication of deductive syllogistic reasoning. Falk and Greenbaum (in press) called this the “illusion of probabilistic proof by contradiction” or the “illusion of attaining improbability.”  [...]   It is the widespread belief that the level of significance at which Ho is rejected, say .05, is the probability that it is correct or, at the very least, that it is of low probability.
The following is almost but not quite the reasoning of null hypothesis rejection:
If the null hypothesis is correct, then this datum (D) can not occur.
It has, however, occurred.
Therefore, the null hypothesis is false.
If this were the reasoning of Ho testing, then it would be formally correct. It would be what Aristotle called the modus tollens, denying the antecedent by denying the
consequent. But this is not the reasoning of NHST. Instead, it makes this reasoning probabilistic, as follows:
If the null hypothesis is correct, then these data are highly unlikely.
These data have occurred.
Therefore, the null hypothesis is highly unlikely.
By making it probabilistic, it becomes invalid. Why? Well, consider this: The following syllogism is sensible and also the formally correct modus tollens:
If a person is a Martian, then he is not a member of Congress.
This person is a member of Congress.
Therefore, he is not a Martian.
Sounds reasonable, no? This next syllogism is not sensible because the major premise is wrong, but the reasoning is as before and still a formally correct modus
tollens:
If a person is an American, then he is not a member of Congress.
(WRONG!)
This person is a member of Congress.
Therefore, he is not an American.
If the major premise is made sensible by making it probabilistic, not absolute, the syllogism becomes formally incorrect and leads to a conclusion that is not sensible:
If a person is an American, then he is probably not a member
of Congress. (TRUE, RIGHT?)
This person is a member of Congress.
Therefore, he is probably not an American. (Pollard &
Richardson. 1987)
This is formally exactly the same as
If Ho is true, then this result (statistical significance) would
probably not occur.
This result has occurred.
Then Ho is probably not true and therefore formally invalid.
This formulation appears at least implicitly in article after article in psychological journals and explicitly in some statistics textbooks—”the illusion of attaining improbability.”

(Quoted from: Jacob Cohen, The Earth Is Round (p < .05), American Psychologist 49, no. 12 (1994), 997-1003.)

About these ads

Responses

  1. I remember liking that paper when I first read it (in fact, you probably gave it to me).

    Surely, though, in a hypothesis test the situation is designed so that H_0 and H_1 are contradictory and H_1 leads to the obtained data being (relatively) likely. The americans/congress example fails this: H_1 here is “the person is not an American”, which actually reduces the probability of the observed outcome (member of congress).

    This is not to assert that the same logical error isn’t widely made in the literature, of course.

  2. So what is the correct conclusion in the probabilistic American/Congress example?

    • The paper quoted contains an explanation. It is less trivial than I would like to have.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

Join 81 other followers

%d bloggers like this: