# Practical implication of failing to reject a null hypothesis

Consider a scenario where you are trying to measure a dosage of a medicine. The machine is calibrated to fill a mean dosage of 50mg. But for a reason you believe that machine's calibration is off. For the above mentioned example null and alternative hypothesis will be as follows:
null: On the average, the dosage sold is 50 mg
alternative: On the average, the dosage sold is not 50 mg
Upon conducting the experiment p-value is greater than say 0.05 for 95% significance level. The conclusion of this test would then be "fail to reject null hypothesis." But practically can I say that the machine is calibrated and is filling a mean dosage of 50mg?

• You are conflation a rejection decision (reject $\text{H}_0$/fail to reject $\text{H}_0$) with drawing a substantive conclusion (i.e. did you find evidence for $\text{H}_{\text{A}}$/fail to find evidence for $\text{H}_{\text{A}}$). These are separate steps in the research process: no one (besides statisticians) gives a hoot about your rejection decision. By contrast your substantive conclusion is the 'juice' (e.g., "We found evidence that/failed to find evidence that the average dose sold is 50mg at the $\alpha=0.5$ level.") Jul 3, 2022 at 17:55
• Whoops! My parenthetical should have read "…the average dose sold is not 50 mg at the $\alpha = 0.5$ level." Jul 3, 2022 at 20:48

Here is a discussion of a simple version of your question. Suppose the dosages are normal. And you have looked at 30 bottles sampled before the faulty filling may have begun and 30 after. Then 'Before' dosages may have been $$X_i \sim\mathsf{Norm}(50, 2),$$ $$i=1,2,\dots,30,$$ and 'After' dosages $$Y_i \sim\mathsf{Norm}(52, 2),$$ $$i =1,2,\dots,30,$$ so that the actual over-filling amount we wish to detect is 2g. Then a Welch two-sample t test will detect the fault in about 97% of such experiments. That is, the power of my proposed test is about 97%, as found by looking at 100,000 simulated Welch t tests below.

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(30, 50, 2),
rnorm(30, 52, 2))\$p.val)
mean(pv <= .05)
[1] 0.96828


Notes: (1) I used the Welch t test because I can only guess whether the standard deviation of the filling amounts is the same before and after the difficulty arose. (2) I am interested in detecting an effect of size 2 (which is the standard deviation of filling amounts). (3) The null hypothesis is rejected if the P-value of the test is smaller than $$0.05.$$ (4) The numeric vector pv contains 100,000 P-values of Welch tests; the logical vector pv <= .05 contains 100,000 TRUEs and FALSEs; and its mean is its proportion of TRUEs.

If you approached the problem this way, then you made a mistake!

What you could have done is calculate the sample size required to detect a difference of practical importance. You know it isn’t filling exactly 50, and you have some tolerable difference from that value. Maybe you can tolerate 49-51, or maybe you can tolerate 49.99-50.01. Then, if you fail to reject the null hypothesis, you know that you adequately powered your investigation to detect important differences and didn’t just fail to reject due to an inadequate sample size (e.g., it’s easy to fail to reject if you make two observations).

This approach requires you to phrase that tolerance in terms of the standard deviation. If this is not viable, then you can do a true equivalence test.

• Alternatively, you could look at the confidence intervals to decide how to interpret the failure to reject the null hypothesis. Jul 3, 2022 at 19:28

This seems like a situation where the failure is relying on null hypothesis significance testing in the first place.

Monitoring that a machine is well calibrated is an estimation problem. You might do better to control accuracy (margin of error) of the dosage rather than the type I error. A type I error in this situation is unnecessarily stopping the production line for maintenance. A type II error is continuing to manufacture medicine with the wrong dosage. The cost of a false negative is much higher than a false positive because, if the dose doesn't match the label, patients could be given incorrect treatment.

This example also seems somewhat naive because drug manufacturing is regulated*. The FDA might not accept the failure to reject a null hypothesis as sufficient evidence of no mis-calibration.