I am reading about decision errors in hypothesis testing. My question is why is a "Type-II error" considered to be an error at all? From what I understand, it arises when we fail to reject a false null hypothesis. When we fail to reject null hypothesis, it simply means that we do not have strong evidence to reject it. We are not making any comment about which of the two hypotheses are true (or false)—either can be true. We are not saying that the null hypothesis is true. Therefore, why is such a conclusion called an error?
It's because we are not doing what we are supposed to do when the alternative hypothesis is true. For example, we are not using the new medicine which is actually better than the existing one but we were unable to prove it.
How we want to use the word "error" is ultimately a semantic issue and reasonable people could disagree on whether, and in what sense, we should consider a false negative to be an error.
- On the one hand, I think you are right that a non-significant result really just means that we do not have enough information to be confident that the null hypothesis is false, and that this does not logically imply that the null hypothesis is true (cf., Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?). So, given the level of confidence that you required in your situation and the level of ambiguity in your data, you made the right decision in the sense of having correctly applied the rule you had decided upon.
- On the other hand, if you put yourself in the position of someone who is planning a study. They want to know if the null is false. If it really is false, they want to walk away after the study is complete having rejected the null hypothesis. Instead, they would remain unclear about the issue and perhaps have to design and run another study. From this perspective, not rejecting a false null is definitely a suboptimal outcome.
The word 'failure' is close to 'error'.
To me the term error makes sense since you can calculate a probability for it to occur (provided you set a certain minimal effect size that would be desirable to detect). And you want to calculate this probability in situations where you want it to be small. In those situations the failure would be considered an error.
To me it is very symmetric with type I errors.
Like p-values, which relate to type I error, you can also calculate the probability for (falsely) not rejecting the null-hypothesis. For a given effect size and a given test (e.g. number of measurements) you can calculate with what probability this 'failure' might occur.
These thoughts do require that you set a boundary for the null hypothesis.
The tendency not to consider type II errors, or at least providing the bounds of the effect size that could have been detected with sufficient probability, is large in a scientific world that is obsessed by p-values, significance, and hypothesis testing (the inverse happens as well by putting large emphasis on minor effects that happened to be significant, only by a huge number of measurements). If $p$ is larger than some $\alpha$ then the effect is said/considered not to be present (or more elegantly not shown to be present). In any way it certainly influences our future actions as if we accept the $H_0$.