1
$\begingroup$

If my main hypothesis is that there will be no difference between two different designs in an experiment, should I be more concerned about the alpha error (rejecting H0 when it is true) or beta error (accepting H0 when it is false)? Also, would my null hypothesis in this experiment be that there is a difference between the two designs?

$\endgroup$
1
  • 1
    $\begingroup$ You need to think about the consequences of being wrong $\alpha$ versus being wrong $\beta$. For example, a false negative ($\beta$ error) is really important when you are diagnosing HIV/AIDS since (a) lack of treatment and (b) transmission are important consequences of a false negative. However, a false positive ($\alpha$ error) in ascertaining which limb a surgeon is amputating is also really important (i.e. loss of healthy limb). Which is more important depends on the substance of your decision to reject or not. $\endgroup$
    – Alexis
    Jul 10, 2020 at 22:13

2 Answers 2

1
$\begingroup$

It depends on the situation. For example, if you work in the medical field and you want to check if the patient has a critical disease like cancer, the false-negative would be terrible. Because if you pass on the sick patient, the patient might die. In this case, you want to reduce the false-negative as much as possible.

One issue is that if you reduce the false positive, the false-negative tends to increase. They work in the opposite way. In many cases, people want a balanced approach reducing two errors at the same time to a reasonable level. And they calculate the scores like F1 which considers both aspects at the same time.


The hypothesis you make is the null hypothesis. In this case, no difference in the designs. And positive means there is a difference against the null hypothesis, and negative is for no difference with a given significance level. Usually, the null hypothesis is set as indistinguishability as it's the simpler hypothesis to test.

As Alexis mentioned in a comment, the hypothesis test doesn't prove or disprove null-hypothesis with certainty. The negation of the null hypothesis doesn't mean that the situation stated by the null hypothesis is impossible. The hypothesis test is about the probabilistic statement.

$\endgroup$
2
  • $\begingroup$ -1 Negative is lack of evidence to reject $H_0$, but is not evidence of no difference. $\endgroup$
    – Alexis
    Jul 10, 2020 at 22:15
  • $\begingroup$ @Alexis Thanks. I updated the answer. $\endgroup$ Jul 11, 2020 at 2:13
3
$\begingroup$

If the hypothesis that you would like to provide support for is "there is no difference bigger than some threshold" (in other words, you want a significant p-value to reflect a time when there is no difference bigger than plus or minus some value), then you may want to look at equivalence testing such as when using two one-sided tests (TOST).

As for errors, you have to balance the "severity" of each of the errors in your context. Lakens has a nice blog post about justifying your alpha which suggests minimizing the combined error rate.

References
Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259-269.

Schuirmann, D. A. (1987). A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6), 657–680.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.