2
$\begingroup$

I know that p-hacking (also know as data dredging) increases the Type I error rate as exemplified by this XKCD example.

My question is whether it influences the Type II error rate. I'm not sure whether it would increase, decrease, or stay the same.

Thanks in advance!

$\endgroup$
0

1 Answer 1

5
$\begingroup$

The Type II error rate is a function of the type I error rate (as well as the sample size and the population parameters - i.e. 'effect size'). So changing the Type I error rate changes the Type II error rate. In particular, if you raise the type I error rate you raise the whole power curve (so the type II error rate goes down).

e.g. see this plot

enter image description here

(taken from my answer here) which shows the impact of moving type I error (betweeb 0.01 and 0.1) on a typical-looking power curve. The Type II error rate is the vertical distance from the curve to the y=1 line (except at the null, which is $δ=0$ in the simple illustrated case for which it is $1-\alpha$), every value of which shrinks when you increase type I error.

There might be other effects, depending on the specific form of p hacking involved.

Nevertheless, nearly always p-hacking reduces type II error. This should not be surprising – after all, the search for significance is usually why people are engaging in it.

In the case where you're testing multiple hypotheses on independent sets of data, where perhaps some nulls might be true and some might be false, obviously the more things you test the better the chance you'll identify at least one of the ones where it's false. In the specific case of testing whether colours of jelly beans might cause acne, there's a very distinct possibility that all nulls are false, in which case you never experience the type II error case (for all that it might be lower if you only could).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.