4
$\begingroup$

Assume we are testing if the true average weight of milk cartons is 100g. We may specify $H_0: \mu = 100$ and $H_1: \mu \ne 100$. Let's assume the true weight is 102.

In the course of testing we may calculate metrics, such as the type 1 error for example. This is the probability that we reject a null hypothesis given that it is true. But isn't this a non-sensical number if the true $\mu$ isn't the same as the null in our test? And given that this is the likely situation in the real world, what information does a hypothesis test really give us if we do not correctly specify the null hypothesis?

$\endgroup$
4
  • 1
    $\begingroup$ It is good to be critical about your null hypothesis, but it is also good to remain practical. We never know the value of the true patametry (if we did then why would we need to do research?). So all we have is the data and some form of hypothesis, and that is all we have to work with. $\endgroup$ – Maarten Buis Sep 17 '20 at 17:46
  • 1
    $\begingroup$ Good to be aware of: for all intents and purposes, the null is almost always false. $\endgroup$ – Firebug Sep 17 '20 at 18:21
  • $\begingroup$ For a point null, yes. For composite nulls, not necessarily. $\endgroup$ – Glen_b Sep 18 '20 at 3:37
  • 1
    $\begingroup$ True @Glen_b (unfortunately we cannot edit comments) $\endgroup$ – Firebug Sep 24 '20 at 23:12
2
$\begingroup$

If the power of your test of $H_0: \mu=100$ against $H_a: \mu\ne 100$ is sufficient, you will likely reject $H_0.$ So the test has not been useless. Furthermore, it is good statistical practice to accompany this test with a CI for $\mu.$ For example, such a CI is included in the R output for t.test.

Also, ideally, the test would have been preceded by a power computation to find the probability of rejection the $H_0$ is false by various amounts $\Delta.$

You are correct that the situation, in which $H_0$ does not exactly specify the true value of $\mu,$ is commonly encountered in practice.

If the variability among contents of milk cartons is given by $\sigma=0.1$ and we sample $n = 12$ cartons, we might get results as shown for the simulated sample below:

set.seed(917)
x = rnorm(12, 102, .1)
t.test(x, mu = 100)

        One Sample t-test

data:  x
t = 66.027, df = 11, p-value = 1.193e-15
alternative hypothesis: 
  true mean is not equal to 100
95 percent confidence interval:
 101.9421 102.0760
sample estimates:
mean of x 
 102.0091 

In this case, $H_0$ is strongly rejected with at P-value very nearly $0.$ The 95% CI $(101.9, 102.1)$ gives a good indication that the true value is near $\mu = 102.$

  • If it is the firm's intention is to overfill cartons slightly in order to avoid complaints or regulatory fines for selling cartons that don't have the $100$g promised on the carton, then result of the experiment and and the test and CI in R will assure them that all is well.

  • If the it is firm's intention to put just barely enough in each carton to avoid underfilling the vast majority of the time, then these results might suggest a target fill amount of something like $100.1$g or $100.2$g, depending on the particulars and pending ongoing monitoring.

Addendum: Because you ask about power computations in a Comment, I will illustrate how one can simulate the power for a two-tailed, one-sample t test, at the 5% level, of $H_0: \mu = 100$ vs. $H_a: \mu = 101$ (specific value different from 100) when $n = 12, \sigma = 1.$ (The result can be found using a noncentral t distribution, but $n$ is too small for a good normal approximation.)

The power is about $88\%.$ That is, when $\mu_a$ differs by $\Delta = 1$ from $\mu_0 = 100,$ we have probability about $0.88$ of rejecting $H_0.$

set.seed(2020)
pv = replicate(10^5, t.test(rnorm(12, 101, 1), mu=100)$p.val)
mean(pv <= 0.05)
[1] 0.88404

The result is essentially the same for this two-tailed test if data are $\mathsf{Norm}(99,1).$ With 100,000 samples of size $n = 12,$ one can expect about 2-place accuracy for rejection probability.

set.seed(1234)
pv = replicate(10^5, t.test(rnorm(12, 99, 1), mu=100)$p.val)
mean(pv <= 0.05)
[1] 0.88219
$\endgroup$
3
  • $\begingroup$ Thanks for this answer, especially the part about the confidence interval, that was very nice. I'd also like to know more about your first sentence that says "If the power of your test is sufficient ,you will likely reject $H_0$. So the test has not been useless". But can't you have a test that is too powerful? Can you give resource that describe how these power calculations are done? $\endgroup$ – Darby Bond Sep 17 '20 at 21:21
  • $\begingroup$ Many mathematical statistics books give a formula for the power of a t test in terms of a 'non-central' t distribution. Intermediate applied texts have approximate formulas for power that are OK for sufficiently large $n$ (using normal approximation). Various statistical programs (R, Minitab, etc.) have 'power and sample size' procedures useful for planning experiments. Also, there are some 'power and sample size calculators' on the Internet (of varying utility and accuracy). For many situations (beyond t tests), closed-form expressions for power are unavailable, and one can use simulation. $\endgroup$ – BruceET Sep 17 '20 at 23:53
  • $\begingroup$ @Darby Bond: "But can't you have a test that is too powerful?" - I added something to this effect to my answer, see there. $\endgroup$ – Lewian Sep 18 '20 at 8:41
2
$\begingroup$

What you call "metrics" are performance characteristics of the test. Regardless of what the true value of $\mu$ is (which we never know), a test that rejects a $H_0$ that is true too often is not good and a rejection is then meaningless. This is what you get from type I error calculations. You also may carry out power calculations. For this you may choose several values of $\mu$ or one borderline value that you "for sure" would like to lead to a significant result. You may also want to know whether the test will likely reject given that the true $\mu$ is not 100 but so close to 100 that you'd consider the $H_0$ still as "practically true" (if not theoretically). (Added after seeing a comment on the other answer:) This can mean that the power of the test is "too high", rejecting the null even in cases in which pragmatically in the real situation there's nothing wrong with it. This as well does not rely on the true $\mu$, because you don't know that, however you can compute whether the test has the performance characteristics given any value of $\mu$ you'd like to try that you expect from it.

Note also that a test does not investigate whether the $H_0$ is true, but rather whether the data are compatible with the $H_0$, i.e., whether they look like typical data generated from the $H_0$. This can well be the case even if the $H_0$ is in fact not true, which means that whatever the true $\mu$ is, the data cannot be used to argue that there's evidence against $H_0$. This (and no more) is what you get from a test.

A last remark: You say the likely situation in the real world is that $\mu$ is not precisely 100. I say it's worse than that. In the real world there is no such thing as a normal distribution, and not even i.i.d. data according to any well defined parametric distribution. There's no such thing as a true distribution, and there's no such thing as a true $\mu$ (which is defined within an assumed model), be it 100, 102 or whatever. Models are thought constructs that help us reasoning about a world that is essentially different. The best thing we can ever do is say, these data look like data generated from an artificial model with a certain parameter value (or a confidence set of parameter values) that has certain characteristics that we may want to interpret.

$\endgroup$
3
  • $\begingroup$ Cogent comments (+1). Given the level of the question, I chose the path of numerical illustrations with numbers similar to those in the question. $\endgroup$ – BruceET Sep 18 '20 at 0:30
  • $\begingroup$ Yes, the question is treated nicely from two quite different perspectives in this way. $\endgroup$ – Lewian Sep 18 '20 at 8:43
  • $\begingroup$ The paragraph on selecting ranges for acceptable $\mu$ was quite illustrative. I think this is something that is neglected in introductory statistics texts $\endgroup$ – Darby Bond Sep 18 '20 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.