The interpretation of hypothesis tests

Home > Subject areas > Studying statistics > The interpretation of hypothesis tests

You need to be very careful in interpreting p values. It is easy to jump to the wrong conclusion.

• The p value in the one card telepathy experiment was 2%. Can we assume this means that the probability of volunteer Y guessing is 2%, so the probability of her being telepathic is 98%? Obviously we can’t! You probably decided that the evidence was not strong enough to convince you that volunteer Y was telepathic, so it certainly doesn’t make sense to say that the probability of her being telepathic is 98%. This probability is the probability of volunteer Y getting the card wrong if she’s guessing – not a very interesting probability. Similarly the 2.5% p value for the second set of exam marks doesn’t mean that the probability of the null hypothesis is 2.5% so the probability that there is a difference between males and females is 97.5%. The p value gives an indication of how plausible the null hypothesis is, but it is not the probability of the null hypothesis being true.

• The second thing to remember is that the p value only tells you about the strength of the evidence. With the first set of exam marks the pvalue was large (53%) indicating that the data was consistent with the null hypothesis. However, this does not prove that there is no difference between males and females in their performance in this exam, just that there is not enough evidence to be certain. On other occasions, you may get a very low p value, suggesting very strong evidence for a difference, but the actual difference may be too small to be of interest. You should always look at the size of any difference, as well as the p value.

There is a strong argument that null hypothesis tests are over-used in many fields of research. Often, alternative approaches give more useful and easily interpreted answers. Two such approaches explained in Making Sense of Statistics are the use of Bayes’ theorem, and confidence intervals.

This content has been written by Michael Wood author of Making Sense of Statistics