Frequentist inference with a null hypothesis that reflects theory a good-enough belt around it

Question

TL;DR:

• With frequentist statistics, does it make sense to 1) no longer use significance testing, 2) set the point null hypothesis to reflect theory and decide a priori when to refute it, and 3) use a "good-enough" belt around this point null hypothesis?
• In R, how can I do this?

This issue is related to Why use a null hypothesis not predicted by theory?

Context

Frequentist statistics is the most widely used statistical method and has many benefits. However, there are several issues with it. I will discuss them briefly here. Then, I will present an example data set, and my question: if my premises are accepted, how should we proceed and analyze these data?

First, frequentist inference usually means null hypothesis significance testing (NHST), and that has been criticized for decades (e.g., Wasserstein et al. 2016, McShane et al. 2019, Amrhein et al. 2019), mainly because it dichotomizes statistics, i.e. results either pass a threshold (almost always p < .05) or not, and results are only taken seriously when they do. This is problematic because significance testing offers a false sense of certainty (true/false), but statistics is hard and there are no shortcuts. The suggestion is to demote p-values from their dominant role, and take all factors into account (e.g., related prior evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain).

Second, scientists are conditioned to test point null hypotheses of no difference (AKA "nil hypotheses"; i.e. $H0: \mu_1 = \mu_2$ and $H1: \mu_1 ≠ \mu_2$). However, the nil hypothesis is known a priori to be false: things are never exactly equal and there is always an effect (Cohen 1994). Two suggestions have been made to improve this, and they can be combined. First, it has been suggested to set up one's theory as the null hypothesis/model, and try to falsify it with empirical evidence (Meehl 1967 called this the "strong" form of hypothesis testing). We do not set an arbitrary threshold (p = .05), but use scientific reasoning to determine, a priori, when we will accept or refute our theory. The second suggestion is to use a "good-enough" belt around our prediction, because the data will never be exactly similar to our expectations (proposed by Serlin & Lapsley, 1985; explained well by Zumbo & Kroc, 2016). Hence, the null hypothesis states that a particular variable, $\delta$, has a particular value, with a good-enough belt of width $\Delta$. By doing so, improvement in experimental method yields stronger corroboration of theory. This let's us arrive at $H0: \delta = \mu ± \Delta$.

Example

The following data are purely hypothetical. Let's say I did a field survey of a plant species, and found it to be more prevalent and larger in clay soils than sandy soils. I hypothesized that growth is similarly affected, so I did a quick pilot. In this pilot, I grew some plants of both species in clay or sand in the lab. I simulated this with the following code:

require(tibble)
require(tidyr)
set.seed(101)
pilot <- tibble(clay = rnorm(n = 10, 15, 4.5), 
                sand = rnorm(n = 10, 12, 2.0)) |> 
  pivot_longer(c("clay", "sand"), 
               names_to = "soil_type", values_to = "seedling_weight")

and found the following seedling weights after 14 days:

| soil_type | mean | sd   |
|-----------|------|------|
| clay      | 16.1 | 2.63 |
| sand      | 11.1 | 1.98 |

This looked promising. Modeling in R as lm(seedling_weight ~ soil_type, data = pilot) shows an intercept of 16.103 and slope of -4.982. So I would use these data to come up with a specific hypothesis, which I would then test with a more comprehensive lab experiment. H0: "after a growth period of 14 days under controlled lab conditions, seedlings of species x grown in clay soil weigh 4.982 ± \Delta grams less than those grown in sandy soil", or $H0: \delta = -4.982 ± \Delta$. I will define $\Delta$ beforehand, but I am not sure how (see below).

So I set up a lab experiment with 40 pots (once again fictional), in which I plant seedlings: half with clay and half with sand. The code to simulate this is as follows:

# I use higher means because growth conditions were probably better in the real experiment
set.seed(101)
experiment <- tibble(clay = rnorm(n = 20, 17, 2.5), 
                sand = rnorm(n = 20, 13, 1.7)) |> 
  pivot_longer(c("clay", "sand"), 
               names_to = "soil_type", values_to = "seedling_weight")

and yields the following seedling weights after 14 days:

| soil_type | mean | sd   |
|-----------|------|------|
| clay      | 16.8 | 2.17 |
| sand      | 13.0 | 1.65 |

Question

I would like to test whether the data from the experiment fall inside or outside the range of the null model $H0: \delta = -4.982 ± \Delta$. Does this make sense? How can I set up this model without using the data from the pilot? (because it might also be derived from theory).

What I'm thinking of is setting $\Delta$ as effect size (suggested by Zumbo & Kroc, 2016), specifically Cohen's d = .5 (large effect). I don't know how to calculate it in this regard. What I would like to know is at what range is a difference noticeable (large). It might for instance be outside -4 and -6. So if my actual experiment turns out outside this range, I will refute my theory and come up with a new hypothesis.

I know of library(infer), where I can test point hypotheses. This is a step in the right direction, because it let's me specify the null model. However, it is much more limited than using direct models (such as lm()), and I still need to figure out $\Delta$ first.

References

• Amrhein, Valentin, Sander Greenland, and Blake McShane. 2019. “Retire Statistical Significance.” Nature 567 (March): 305–7.https://doi.org/10.1080/00031305.2018.1527253.
• Cohen, Jacob. 1994. “The Earth Is Round (p < .05).” American Psychologist 49 (12): 997–1003. https://doi.org/10.1037//0003-066X.49.12.997.
• McShane, Blakeley B., David Gal, Andrew Gelman, Christian Robert, and Jennifer L. Tackett. 2019. “Abandon Statistical Significance.” The American Statistician 73 (sup1): 235–45.
• Meehl, Paul E. 1967. “Theory-Testing in Psychology and Physics: A Methodological Paradox.” Philosophy of Science 34 (2): 103–15.
• Serlin, Ronald C., and Daniel K. Lapsley. 1985. “Rationality in Psychological Research: The Good-Enough Principle.” American Psychologist 40 (1): 73–83. https://doi.org/10.1037/0003-066X.40.1.73.[https://doi.org/10.1086/288135](https://doi.org/10.1086/288135).
• Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA’s Statement on p -Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108
• Zumbo, Bruce D., and Edward Kroc. 2016. “Some Remarks on Rao and Lovric’s ‘Testing the Point Null Hypothesis of a Normal Mean and the Truth: 21st Century Perspective.’” Journal of Modern Applied Statistical Methods 15 (2): 33–40. https://doi.org/10.22237/jmasm/1478001780.

Answer 1

I agree that the null hypothesis of equivalence is, in many cases, a rather useless hypothesis. In such cases, a superiority hypothesis informed by theory/other empirical results may be preferred. However, I don't see the need for a new procedure here. I'd suggest you 1) set a superiority hypothesis and 2) use a t-test to decide whether or not to reject this hypothesis.

So your hypothesis would be as follows:

$$ H_0: \hat \mu_t - \hat \mu_c \le \delta \\ H_1: \hat \mu_t - \hat \mu_c > \delta $$

And then your test as follows:

$$ \frac{\hat \mu_t - \hat \mu_c - \delta} {s \sqrt{\frac{1}{n_t} + \frac{1}{n_c}}} > t_{\alpha}, \ n_t + n_c - 2 $$

Answer 2

As far as I can see, your main question can be addressed by the answer of num_39 or probably also by a confidence interval (maybe one-sided).

I will address some other issues raised in the question. I think that it is very important to distinguish between the formal concept of a significance test (and p-value) and the way how it is interpreted (often misinterpreted). There is a tendency in some current literature criticising significance tests to blame misinterpretations on the concept itself, but in my view the concept itself can be used in a valid and unproblematic way, whereas what needs to be criticised is its widespread misinterpretation and misuse. This is to some extent caused by its own success, because at some point many journal editors, reviewers etc. made it explicitly or implicitly mandatory for publications to come up with significant results, which created a very unhealthy incentive for trying to tease significance out of anything. I give to the opponents of significance testing that it is legitimate to wonder whether some problems are to some extent intrinsic to the significance test concept itself, however I am rather a member of the "don't throw the baby out with the bathwater" fraction.

Ultimately the idea of significance testing is very old and quite intuitive. It basically says that data provide evidence against a probability model if the data are very unlikely under that model. We need to have in mind here that many probability distributions provide nonzero probability to everything that can possibly happen, so one can't learn enough about whether a probability model is appropriate or not by only rejecting it if something under the model impossible happens. Furthermore, things are complicated by the fact that continuous probability distributions will give probability zero to any precise result, so that in principle one could declare any data as "very unlikely", which isn't helpful either. This means that in order to find evidence against a model, one need to specify a set of events pre-data that has a probability deemed too small, and basically state that the model doesn't predict this set to occur, and if it occurs, this constitutes evidence against the model (the role of the alternative hypothesis is that having an alternative in mind helps to choose the set in such a way that, if indeed the model is wrong in a certain way suspected a priori, the "rejecting set" is likely to occur, i.e., we have a good chance to reject a false model in case our suspected alternative is true). In my view this is essentially the most direct way to make a statement about whether data are compatible with the model.

Some comments:

1. A test can formally be defined as binary, i.e., either "rejecting" or "accepting" the null hypothesis (the latter is a terrible term as there are always many models compatible with any data, and therefore we cannot have evidence in favour of any specific null hypothesis to be true). However, it should be clear that this oversimplifies the situation, as for defining a binary decision rule we need to choose a probability threshold, and exact probability thresholds are artificial. Is 0.07 a so small probability that it should be taken as a reason to "reject" the model? 0.04? 0.015? 0.0099? There is no objective answer to this, and in fact it doesn't need to be decided unless there are different actions involved whether the outcome is one or the other. p-values are meant to give "continuous" information rather than a binary decision rule, and everybody understanding p-values understands that 0.04 and 0.06 are in fact more similar to each other than either of these is to 0.2 or 0.001 even though somebody may put a threshold for action at 0.05. It needs to be understood also that if binary decisions are to be made, thresholds are required, and if we can't have objectively justified ones, whatever we do will come with a smell of arbitrariness. (Note that in some literature multiple thresholds are used for talking about "weak/modest/strong/very strong evidence" - this gives more information than "reject/don't reject" but less than the continuous p-value.) The question states "significance testing offers a false sense of certainty (true/false)", however who understands the nature of the decision problem should know that certainty is not provided because (a) any binary decision has to depend on an at least to some extent arbitrary threshold and (b) we are rejecting or not rejecting abstract formal models, and reality will be different from the model in some ways anyway.

2. Personally I don't think there is any such thing as a "true" model in reality. Probability models are defined in the world of mathematics, which is essentially different from the reality for which it is interpreted. Models are tools for thinking, and no test can say anything about the "truth" of a model. This implies in particular that the null hypotheses should not be believed to be "true" regardless of whether the data reject it or not, and the same holds for the alternative (and any probability model used in other approaches outside the significance test paradigm). Testing the null hypothesis does not mean that we are testing whether it is literally true, but rather whether data are incompatible with it, for which reason we may drop it, not as a "belief" (because I wouldn't believe it in the first place), but even as a tool, a means to understand and interpret reality. This means that the following is mistaken as objection against significance tests: "However, the nil hypothesis is known a priori to be false: things are never exactly equal and there is always an effect." Even though I agree with the null hypothesis never being true (which is not exclusive to the zero effect hypothesis, rather also to any precisely specified effect), this is not what the test is meant to find out. For example, if an astrologist claims that marriages between certain zodiac signs are more likely to fail than others, it may well be that observed data are consistent with total randomness (not sure whether Cohen would claim that there for sure is an effect in this situation, in fact I can imagine reasons for it such as people asking an astrologist for advice who says "get divorced"), and it is a legitimate interest whether they are or not. (If you don't believe in astrology, it may still be of interest whether astrology talk has this kind of influence on society. Before having seen data, I don't have a strong expectation either way, so it is an interesting question what the outcome is.) Ultimately, if somebody claims that there is a certain kind of effect, this claim is for sure weakened by realising that data are compatible with randomness or a nil effect. This of course is totally perverted by a culture that gives scientists an incentive to do something that seems to "prove" whatever claim they may have, so that something will be done in order to achieve significance that a neutral tester of the claim wouldn't achieve with high probability (and also, as stated before, rejecting a null hypothesis does not imply that any specific alternative is true).

3. "scientists are conditioned to test point null hypotheses of no difference" - there is nothing in the formal concept of significance tests that requires this, even though, as stated above, in some situations it can make sense. The issue mentioned above, namely that models are essentially different from reality, therefore never literally (and precisely) true, holds as well for other point null hypotheses by the way, such as the one of a "meaningful minimum distance" in the response by num_39. This doesn't make tests useless, as long as before collecting the data it is really of interest whether data will show evidence against the H0 or not (which is of course different from the situation that a researcher is determined to find such evidence whatever it takes). Note by the way that the objection applies less strongly to one-sided tests; part of the H0 can then be not only a nil effect but also all effects that go in the other direction than what is expected, which in practice happens from time to time.

4. It is true that a test outcome in itself doesn't say anything about effect sizes, and that effect sizes are usually relevant. It is of course a misconception of a significance test to think that the test decides whether an effect should be taken as substantively meaningful. Some people seem to think that this should be a reason to not run significance tests at all, or to replace them, e.g., by confidence intervals or Bayesian analysis. In my view it'd be so much easier to acknowledge that different methods are for different kinds of questions, sometimes effect sizes are the major focus of interest but sometimes the compatibility of data with a null model, and sometimes both (or other things such as prediction quality). Whether tests (and/or other methods) should be used or not depends on the question of interest, and for sure if you run a test, it doesn't mean you are not allowed to compute a confidence interval on top of it, or even a Bayesian analysis!

5. Whether point null hypotheses should be used to reflect theory obviously depends on whether the theory allows such precise specifications.

All of this doesn't seem to be that mysterious to me. Much discussion on significance tests in my view treats them as some kind of black magic that is expected to deliver all kinds of miracles, and is then condemned for not doing so. I don't think they are very problematic if they are used for what they can do, and not used for what they can't do.