Statistical Significance or Unambiguous Direction of Influence?

Question

TL; DR

In the context of a linear regression model, we run a statistical test for whether an estimated coefficient is "statistically significant". We will say that it is if we reject the null of it being zero, for a given type I error.

One can verify that:

The null hypothesis will be rejected (=> the coefficient is "statistically significant") if and only if the corresponding confidence interval for the coefficient contains only positive, or only negative, values.

Equivalently,
The null hypothesis will not be rejected if and only if the corresponding confidence interval for the coefficient contains both positive and negative values.

The above seems to say that "statistical significance" is mathematically equivalent to "unambiguous direction of influence" on (or of covariance with) the dependent variable, given the chosen type I error. In other words, the variance relative to the magnitude of the estimate is so great (=> estimation uncertainty is so big) that we cannot even say what the sign of the coefficient will likely be.

An ambiguous direction of influence/covariance seems pretty unmanageable from a logical point of view, and useless from a practical point of view. When discussing estimation results, what is the value of saying "the effect can be negative, but it can also be positive"?

Q: Do you know of cases/literature where "statistical significance" is discussed from this perspective, i.e. as a minimum necessary condition to be able to say something useful about an estimate?

TS; DR

With this post I want to put forward an interpretation of the concept of "Statistical Significance" in the context of "frequentist" Hypothesis testing that rings very convincing to my ears. I am not arguing that it is the "true" or "correct" interpretation, I am not seeing it as an antagonistic interpretation to any other.

I will not enter into the methodological debates or critiques of these tests. I will just accept them as they are, and I will try to explain how I perceive their results so that they help me in my reasoning. Naturally, I have the hope that it may also appear convincing and helpful for some of the members of this community, and this is why I am writing this.

Since this is a Q&A site, my question(s) are:

What are the conceptual, methodological, logical flaws, gaps, neglected aspects, in my interpretational argument? (on the side, I of course wish that you will share your opinion too, but beware, answers here should not be "primarily opinion based"!).

Also, there is a strictly positive probability that I am re-inventing the wheel here, so Can you point to literature where this interpretation has already appeared?

The Case
Consider the most basic "statistical significance" test in econometrics, the two-sided t-test on an estimated regression coefficient $\hat \beta$ with standard error $SE(\hat \beta)$ , where in order to test the null hypothesis that this coefficient is "statistically insignificant", we form the ratio $\hat \beta/SE(\hat \beta)$ and, given an exogenously chosen Type I error probability denoted by $\alpha$, we characterize the coefficient as "statistically significant" if

$$\left|\frac{\hat \beta}{SE(\hat \beta)}\right| \geq T\left(n-k, 1-\frac {\alpha}2\right)$$,

where the right hand side of the inequality is the value of Student’s t cumulative distribution function (cdf) for $n – k$ degrees of freedom ($n$ being the sample size and $k$ being the number of regressors) at the point $1-\frac {\alpha}2$. If degrees of freedom are "many", the standard normal cdf may be used instead.
Now consider the corresponding confidence interval:

$$CI(\hat \beta\mid \alpha) = \hat \beta \pm SE(\hat \beta)\cdot T\left(n-k, 1-\frac {\alpha}2\right)$$

At the threshold for "statistical significance", where $\left|\hat \beta/SE(\hat \beta)\right| = T\left(n-k, 1-\frac {\alpha}2\right)$, the corresponding confidence interval is always equal to $[0,2\hat \beta]$ (or $[2\hat \beta, 0]$ if the point estimate is negative), for any $\alpha$, any size of type I error probability such that we are at the threshold.

So, if $\alpha$ is such that $\left|\hat \beta/SE(\hat \beta)\right| < T\left(n-k, 1-\frac {\alpha}2\right)$ the coefficient will be characterized as "statistically insignificant", while at the same time, the corresponding CI will always include the possibility of a sign reversal.

Equivalently, for any chosen Type I error probability, the corresponding confidence interval for a coefficient accepted as "statistically significant" will never contain a sign reversal.

The "difference in means" statistical test falls also in the same category.

I have read phrases like "if statistically insignificant, then the confidence interval will include the value zero and so the possibility that the coefficient is zero" -but who really cares about a single point-value of a continuous random variable? But even if it is non-continuous, a non-zero probability of being zero is just that -one out of many probable outcomes.

The Interpretation
A sign reversal means the possibility of reversal in the direction of influence, and this is a situation that we cannot really accommodate. So in my eyes, "Statistical significance" can also be viewed as a much better-sounding misnomer for "non-ambiguity in the sign" (always in a probabilistic sense of course). If the point estimate is probabilistically sign-ambiguous, what can we usefully say about the relation between the dependent variable and the regressor under discussion, since the coefficient reflecting this relation can be positive, but it can also be negative? Resolution (probabilistically) of this qualitative feature of the relationship is a necessary step prior to any meaningful quantitative assessment.

Under this light, "Statistical Significance" is not some major finding: it is the barest minimum requirement in order to keep into the conversation the quantitative results produced by the estimation procedure on the data set. If "statistically insignificant", these results appear not really usable, in any logically coherent and consistent way.

This is of course an interpretation given that we accept the results of the Hypothesis testing methodology, and the methodology itself. So I do not touch on the issue of whether these results are misleading due to any kind of misspecification, technical issues etc, or of whether Hypothesis testing is fundamentally flawed. I am just laying down a way to interpret what "statistical significance" can ...signify (probabilistically unambiguous direction of influence), given that we accept the related framework in which it emerges as a legitimate and valid tool.

Related CV posts could be
https://stats.stackexchange.com/questions/72782/going-from-rejecting-the-null-to-inferring-the-sign-of-the-population-parameter

How to quantify statistical insignificance?

Can a narrow confidence interval around a non-significant effect provide evidence for the null?

Does statistically insignificant difference of means imply equality of means?

Is statistical insignificance fatal?

Answer 1

I think this way about the relationship between NHSTs and CIs in general, but don't know of any references that describe everything the same way off the top of my head...Seems there's bound to be some out there though, as resolving directional ambiguity of an effect is the most compelling reason to perform a NHST that I know of.

However, statistical significance doesn't really truly completely absolutely resolve this ambiguity. As you say (I'm sure most/all of this has been said before), the resolution is only probabilistic, and that probability is $1-\alpha$ in the Neyman–Pearson framework or $1-p$ in the Fisherian framework (AFAIK; I could be misrepresenting the latter). Thus it isn't even a particularly foolproof "barest minimum requirement". The choice of $\alpha$ is at worst arbitrary, usually a matter of scientific convention, and at best a principled and measured departure from convention (e.g., "I have lots of power, and a false alarm error would cause major harm").

Being a matter of convention isn't terrible, but it isn't quite foolproof either, and causes some known problems for the big picture because we give it so many chances to go wrong. Consider how easily people lose sight of the non-finality of a result, and how much hinges on crossing that threshold. Arguably, this is the flaw in your interpretation: one could still make practical use of barer minimums. People often choose to replicate studies with $p=.06$, and if they can afford to, why shouldn't they? These people should probably choose $\alpha>.05$, or multiple $\alpha$s corresponding to different decisions:

• E.g., $p<\alpha_1=.04$: reject $\rm H_0$ comfortably and proceed to build on $\rm H_A$, ideally while continuing to gather replicative evidence
• $\alpha_1\le p<\alpha_2\approx[.15,.30]$ (somewhere in there depending on how willing one is to replicate): interpret with caution and seek replication before building on $\rm H_A$
• $p>\alpha_2$, given sufficient power: abandon hope of rejecting $\rm H_0$ meaningfully and reevaluate life choices – unless the null result is actually useful, of course.

Power is tied inextricably into the meaning of NHST results too...but maybe these are all the issues you intended to set aside by stipulating provisional acceptance of frequentist NHST methodology. I admit, the Neyman–Pearson $\alpha$ may even be optimal – if not foolproof – as a choice of general, if simplistic statistical practice. Some simplification of the interpretive process probably helps broaden the accessibility of inferential statistics, which might only be more esoteric and still no less error-prone in general practice if tests were interpreted less dichotomously as Fisher seems to have intended. I'm a fan of confidence intervals in general, and might be doing them a disservice by admitting that Neyman–Pearson NHST might deserve its place in entry-level / mainstream statistics, but my only issues (for now) with your interpretation are my issues with NHST in general.

Oh, and another "gap": this doesn't really apply to nulls other than zero. Granted, those are unusual; the power of convention (or the fear of defying reviewers' expectations) is such that probably more than 95% of analysts stick to both $\alpha=.05$ and $\rm H_0:\mu=0$...but this isn't mandatory, and I often find myself reminding people of that. The adjustment to what you've said that's necessary to accommodate other nulls isn't hard though. It would just be a question of whether the effect is probably ($1-\alpha$) in the hypothesized direction and of at least the hypothesized size.