- With frequentist statistics, does it make sense to 1) no longer use significance testing, 2) set the point null hypothesis to reflect theory and decide a priori when to refute it, and 3) use a "good-enough" belt around this point null hypothesis?
- In R, how can I do this?
This issue is related to Why use a null hypothesis not predicted by theory?
Frequentist statistics is the most widely used statistical method and has many benefits. However, there are several issues with it. I will discuss them briefly here. Then, I will present an example data set, and my question: if my premises are accepted, how should we proceed and analyze these data?
First, frequentist inference usually means null hypothesis significance testing (NHST), and that has been criticized for decades (e.g., Wasserstein et al. 2016, McShane et al. 2019, Amrhein et al. 2019), mainly because it dichotomizes statistics, i.e. results either pass a threshold (almost always p < .05) or not, and results are only taken seriously when they do. This is problematic because significance testing offers a false sense of certainty (true/false), but statistics is hard and there are no shortcuts. The suggestion is to demote p-values from their dominant role, and take all factors into account (e.g., related prior evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain).
Second, scientists are conditioned to test point null hypotheses of no difference (AKA "nil hypotheses"; i.e. $H0: \mu_1 = \mu_2$ and $H1: \mu_1 ≠ \mu_2$). However, the nil hypothesis is known a priori to be false: things are never exactly equal and there is always an effect (Cohen 1994). Two suggestions have been made to improve this, and they can be combined. First, it has been suggested to set up one's theory as the null hypothesis/model, and try to falsify it with empirical evidence (Meehl 1967 called this the "strong" form of hypothesis testing). We do not set an arbitrary threshold (p = .05), but use scientific reasoning to determine, a priori, when we will accept or refute our theory. The second suggestion is to use a "good-enough" belt around our prediction, because the data will never be exactly similar to our expectations (proposed by Serlin & Lapsley, 1985; explained well by Zumbo & Kroc, 2016). Hence, the null hypothesis states that a particular variable, $\delta$, has a particular value, with a good-enough belt of width $\Delta$. By doing so, improvement in experimental method yields stronger corroboration of theory. This let's us arrive at $H0: \delta = \mu ± \Delta$.
The following data are purely hypothetical. Let's say I did a field survey of a plant species, and found it to be more prevalent and larger in clay soils than sandy soils. I hypothesized that growth is similarly affected, so I did a quick pilot. In this pilot, I grew some plants of both species in clay or sand in the lab. I simulated this with the following code:
require(tibble) require(tidyr) set.seed(101) pilot <- tibble(clay = rnorm(n = 10, 15, 4.5), sand = rnorm(n = 10, 12, 2.0)) |> pivot_longer(c("clay", "sand"), names_to = "soil_type", values_to = "seedling_weight")
and found the following seedling weights after 14 days:
| soil_type | mean | sd | |-----------|------|------| | clay | 16.1 | 2.63 | | sand | 11.1 | 1.98 |
This looked promising. Modeling in R as
lm(seedling_weight ~ soil_type, data = pilot) shows an intercept of 16.103 and slope of -4.982. So I would use these data to come up with a specific hypothesis, which I would then test with a more comprehensive lab experiment. H0: "after a growth period of 14 days under controlled lab conditions, seedlings of species x grown in clay soil weigh 4.982 ± \Delta grams less than those grown in sandy soil", or $H0: \delta = -4.982 ± \Delta$. I will define $\Delta$ beforehand, but I am not sure how (see below).
So I set up a lab experiment with 40 pots (once again fictional), in which I plant seedlings: half with clay and half with sand. The code to simulate this is as follows:
# I use higher means because growth conditions were probably better in the real experiment set.seed(101) experiment <- tibble(clay = rnorm(n = 20, 17, 2.5), sand = rnorm(n = 20, 13, 1.7)) |> pivot_longer(c("clay", "sand"), names_to = "soil_type", values_to = "seedling_weight")
and yields the following seedling weights after 14 days:
| soil_type | mean | sd | |-----------|------|------| | clay | 16.8 | 2.17 | | sand | 13.0 | 1.65 |
I would like to test whether the data from the experiment fall inside or outside the range of the null model $H0: \delta = -4.982 ± \Delta$. Does this make sense? How can I set up this model without using the data from the pilot? (because it might also be derived from theory).
What I'm thinking of is setting $\Delta$ as effect size (suggested by Zumbo & Kroc, 2016), specifically Cohen's d = .5 (large effect). I don't know how to calculate it in this regard. What I would like to know is at what range is a difference noticeable (large). It might for instance be outside -4 and -6. So if my actual experiment turns out outside this range, I will refute my theory and come up with a new hypothesis.
I know of
library(infer), where I can test point hypotheses. This is a step in the right direction, because it let's me specify the null model. However, it is much more limited than using direct models (such as
lm()), and I still need to figure out $\Delta$ first.
- Amrhein, Valentin, Sander Greenland, and Blake McShane. 2019. “Retire Statistical Significance.” Nature 567 (March): 305–7.https://doi.org/10.1080/00031305.2018.1527253.
- Cohen, Jacob. 1994. “The Earth Is Round (p < .05).” American Psychologist 49 (12): 997–1003. https://doi.org/10.1037//0003-066X.49.12.997.
- McShane, Blakeley B., David Gal, Andrew Gelman, Christian Robert, and Jennifer L. Tackett. 2019. “Abandon Statistical Significance.” The American Statistician 73 (sup1): 235–45.
- Meehl, Paul E. 1967. “Theory-Testing in Psychology and Physics: A Methodological Paradox.” Philosophy of Science 34 (2): 103–15.
- Serlin, Ronald C., and Daniel K. Lapsley. 1985. “Rationality in Psychological Research: The Good-Enough Principle.” American Psychologist 40 (1): 73–83. https://doi.org/10.1037/0003-066X.40.1.73.[https://doi.org/10.1086/288135](https://doi.org/10.1086/288135).
- Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA’s Statement on p -Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108
- Zumbo, Bruce D., and Edward Kroc. 2016. “Some Remarks on Rao and Lovric’s ‘Testing the Point Null Hypothesis of a Normal Mean and the Truth: 21st Century Perspective.’” Journal of Modern Applied Statistical Methods 15 (2): 33–40. https://doi.org/10.22237/jmasm/1478001780.