I am reading about decision errors in hypothesis testing. My question is why is a "Type-II error" considered to be an error at all? From what I understand, it arises when we fail to reject a false null hypothesis. When we fail to reject null hypothesis, it simply means that we do not have strong evidence to reject it. We are not making any comment about which of the two hypotheses are true (or false)—either can be true. We are not saying that the null hypothesis is true. Therefore, why is such a conclusion called an error?
$\begingroup$
$\endgroup$
0
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
3
It's because we are not doing what we are supposed to do when the alternative hypothesis is true. For example, we are not using the new medicine which is actually better than the existing one but we were unable to prove it.
-
$\begingroup$ Going by this logic, failing to reject a true null hypothesis should not be a true positive. It should be an error because we are not doing what we are supposed to do when the null hypothesis is true. $\endgroup$ Commented Jun 28, 2017 at 22:07
-
1$\begingroup$ Why do you say this? Of course we are doing what we are supposed to do when the null hypothesis is true - we are staying with the current behavior, which is the correct thing to do under your assumption that the null hypothesis is true. $\endgroup$ Commented Jun 30, 2017 at 14:44
-
1$\begingroup$ Connecting NHST with the actions that typically ensue, as Zahava Kor does here, seems the simplest and most direct way to appreciate Type II errors as 'errors'. In some contexts, such as that of drug development which Zahava Kor cites specifically, the NHST is embedded within a plan of action such that the epistemic outcome of the test determines actions. In connection with medicine generally, it helps to appreciate Braithwaite's magnificent EBM’s Six Dangerous Words JAMA. 2013;310(20):2149-2150. doi:10.1001/jama.2013.281996. $\endgroup$ Commented Jul 4, 2017 at 19:46
$\begingroup$
$\endgroup$
1
How we want to use the word "error" is ultimately a semantic issue and reasonable people could disagree on whether, and in what sense, we should consider a false negative to be an error.
- On the one hand, I think you are right that a non-significant result really just means that we do not have enough information to be confident that the null hypothesis is false, and that this does not logically imply that the null hypothesis is true (cf., Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?). So, given the level of confidence that you required in your situation and the level of ambiguity in your data, you made the right decision in the sense of having correctly applied the rule you had decided upon.
- On the other hand, if you put yourself in the position of someone who is planning a study. They want to know if the null is false. If it really is false, they want to walk away after the study is complete having rejected the null hypothesis. Instead, they would remain unclear about the issue and perhaps have to design and run another study. From this perspective, not rejecting a false null is definitely a suboptimal outcome.
answered Jun 28, 2017 at 18:56
-
3$\begingroup$ +1. But it might be appropriate to bring up the issue of Fisher vs Neyman-Pearson here (I will link to your own nice answer about it: stats.stackexchange.com/questions/23142). OP is asking the question in what seems to be Fisherian paradigm. But Type II error is a concept from Neyman-Pearson paradigm. Hence, I believe, OP's confusion. $\endgroup$– amoebaCommented Jun 29, 2017 at 7:26
$\begingroup$
$\endgroup$
1
The word 'failure' is close to 'error'.
To me the term error makes sense since you can calculate a probability for it to occur (provided you set a certain minimal effect size that would be desirable to detect). And you want to calculate this probability in situations where you want it to be small. In those situations the failure would be considered an error.
To me it is very symmetric with type I errors.
Like p-values, which relate to type I error, you can also calculate the probability for (falsely) not rejecting the null-hypothesis. For a given effect size and a given test (e.g. number of measurements) you can calculate with what probability this 'failure' might occur.
These thoughts do require that you set a boundary for the null hypothesis.
The tendency not to consider type II errors, or at least providing the bounds of the effect size that could have been detected with sufficient probability, is large in a scientific world that is obsessed by p-values, significance, and hypothesis testing (the inverse happens as well by putting large emphasis on minor effects that happened to be significant, only by a huge number of measurements). If $p$ is larger than some $\alpha$ then the effect is said/considered not to be present (or more elegantly not shown to be present). In any way it certainly influences our future actions as if we accept the $H_0$.
answered Jun 30, 2017 at 10:49
-
1$\begingroup$ Actually, the real error is doing hypothesis testing. $\endgroup$ Commented Jun 30, 2017 at 11:03