Is a hypothesis test useful if our null hypothesis is not the true value?

Question

Assume we are testing if the true average weight of milk cartons is 100g. We may specify $H_0: \mu = 100$ and $H_1: \mu \ne 100$. Let's assume the true weight is 102.

In the course of testing we may calculate metrics, such as the type 1 error for example. This is the probability that we reject a null hypothesis given that it is true. But isn't this a non-sensical number if the true $\mu$ isn't the same as the null in our test? And given that this is the likely situation in the real world, what information does a hypothesis test really give us if we do not correctly specify the null hypothesis?

Answer 1

If the power of your test of $H_0: \mu=100$ against $H_a: \mu\ne 100$ is sufficient, you will likely reject $H_0.$ So the test has not been useless. Furthermore, it is good statistical practice to accompany this test with a CI for $\mu.$ For example, such a CI is included in the R output for t.test.

Also, ideally, the test would have been preceded by a power computation to find the probability of rejection the $H_0$ is false by various amounts $\Delta.$

You are correct that the situation, in which $H_0$ does not exactly specify the true value of $\mu,$ is commonly encountered in practice.

If the variability among contents of milk cartons is given by $\sigma=0.1$ and we sample $n = 12$ cartons, we might get results as shown for the simulated sample below:

set.seed(917)
x = rnorm(12, 102, .1)
t.test(x, mu = 100)

        One Sample t-test

data:  x
t = 66.027, df = 11, p-value = 1.193e-15
alternative hypothesis: 
  true mean is not equal to 100
95 percent confidence interval:
 101.9421 102.0760
sample estimates:
mean of x 
 102.0091 

In this case, $H_0$ is strongly rejected with at P-value very nearly $0.$ The 95% CI $(101.9, 102.1)$ gives a good indication that the true value is near $\mu = 102.$

• If it is the firm's intention is to overfill cartons slightly in order to avoid complaints or regulatory fines for selling cartons that don't have the $100$g promised on the carton, then result of the experiment and and the test and CI in R will assure them that all is well.

• If the it is firm's intention to put just barely enough in each carton to avoid underfilling the vast majority of the time, then these results might suggest a target fill amount of something like $100.1$g or $100.2$g, depending on the particulars and pending ongoing monitoring.

Addendum: Because you ask about power computations in a Comment, I will illustrate how one can simulate the power for a two-tailed, one-sample t test, at the 5% level, of $H_0: \mu = 100$ vs. $H_a: \mu = 101$ (specific value different from 100) when $n = 12, \sigma = 1.$ (The result can be found using a noncentral t distribution, but $n$ is too small for a good normal approximation.)

The power is about $88\%.$ That is, when $\mu_a$ differs by $\Delta = 1$ from $\mu_0 = 100,$ we have probability about $0.88$ of rejecting $H_0.$

set.seed(2020)
pv = replicate(10^5, t.test(rnorm(12, 101, 1), mu=100)$p.val)
mean(pv <= 0.05)
[1] 0.88404

The result is essentially the same for this two-tailed test if data are $\mathsf{Norm}(99,1).$ With 100,000 samples of size $n = 12,$ one can expect about 2-place accuracy for rejection probability.

set.seed(1234)
pv = replicate(10^5, t.test(rnorm(12, 99, 1), mu=100)$p.val)
mean(pv <= 0.05)
[1] 0.88219

Answer 2

What you call "metrics" are performance characteristics of the test. Regardless of what the true value of $\mu$ is (which we never know), a test that rejects a $H_0$ that is true too often is not good and a rejection is then meaningless. This is what you get from type I error calculations. You also may carry out power calculations. For this you may choose several values of $\mu$ or one borderline value that you "for sure" would like to lead to a significant result. You may also want to know whether the test will likely reject given that the true $\mu$ is not 100 but so close to 100 that you'd consider the $H_0$ still as "practically true" (if not theoretically). (Added after seeing a comment on the other answer:) This can mean that the power of the test is "too high", rejecting the null even in cases in which pragmatically in the real situation there's nothing wrong with it. This as well does not rely on the true $\mu$, because you don't know that, however you can compute whether the test has the performance characteristics given any value of $\mu$ you'd like to try that you expect from it.

Note also that a test does not investigate whether the $H_0$ is true, but rather whether the data are compatible with the $H_0$, i.e., whether they look like typical data generated from the $H_0$. This can well be the case even if the $H_0$ is in fact not true, which means that whatever the true $\mu$ is, the data cannot be used to argue that there's evidence against $H_0$. This (and no more) is what you get from a test.

A last remark: You say the likely situation in the real world is that $\mu$ is not precisely 100. I say it's worse than that. In the real world there is no such thing as a normal distribution, and not even i.i.d. data according to any well defined parametric distribution. There's no such thing as a true distribution, and there's no such thing as a true $\mu$ (which is defined within an assumed model), be it 100, 102 or whatever. Models are thought constructs that help us reasoning about a world that is essentially different. The best thing we can ever do is say, these data look like data generated from an artificial model with a certain parameter value (or a confidence set of parameter values) that has certain characteristics that we may want to interpret.