7
$\begingroup$

I'm working on a regression analysis and have obtained a point estimate that is statistically non-significant. Economically, a non-significant result makes sense in my context, but I want to ensure that this finding isn’t due to a lack of power. Instead, I want to confirm that the effect is indeed close to zero and not just statistically non-significant.

I’m aiming to make statements like, "... From the confidence intervals, we can rule out effects larger than 1–1.4 months of increased life expectancy," as done by Meghir, Palme, & Simeonova (2018), or, "The effects can usually be bounded to a tight interval around zero," as mentioned by Cesarini et al. (2016).

Given my point estimate and its confidence interval, how can I confidently interpret the results to make a similar statement? Specifically, how do I quantify the precision of my estimate to rule out large effects?

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ Not sure what else you expect here than saying that what is not in the confidence interval seems to be at odds with the data (which in the cited literature is apparently worded as "we can rule it out")? $\endgroup$
    – Christian Hennig
    Commented Sep 5 at 9:04
  • $\begingroup$ I was thinking about something like ruling out large effects given a point estimate and a CI or using the notion of tight intervals. I am in Econ and it is not easy to sell a null result, especially just saying it's not in the CI. $\endgroup$
    – PostDocing
    Commented Sep 5 at 9:07
  • $\begingroup$ Are you saying your "null result" is not in the CI? There is a correspondence between tests and CIs, so normally (unless you use a CI that isn't properly adapted to your test) if you don't reject the H0, it should be in the CI. If it isn't, chances are it would help us if you explain in detail what you did and what the results are. $\endgroup$
    – Christian Hennig
    Commented Sep 5 at 9:12
  • 2
    $\begingroup$ By the way, "a non-significant result makes sense in my context, but I want to ensure that this finding isn’t due to a lack of power". In general you can't do that, see stats.stackexchange.com/questions/578478/… $\endgroup$
    – Christian Hennig
    Commented Sep 5 at 9:15
  • 3
    $\begingroup$ The CI is a set of values compatible with the data. So you can say what is outside the CI is incompatible with the data. $\endgroup$
    – Christian Hennig
    Commented Sep 5 at 9:16

3 Answers 3

Reset to default
10
$\begingroup$

One thing that you may want to look at are equivalence tests (TOST). Here the standard way of running tests is reversed and you test the null hypothesis that an effect has a certain minimum size against the alternative that it is closer to zero. In such tests, if you reject, you can say that you have evidence that the parameter is so-and-so close to zero.

Improve this answer
$\endgroup$
2
  • 4
    $\begingroup$ In this case, the OP already ran a "standard" two sided-test because their preference was to get a significant result. If based on this first analysis, the OP decides to run an equivalence test, then isn't that similar in spirit to a two-stage procedure similar to: Citations explaining the problems with two-stage statistical testing? $\endgroup$
    – dipetkov
    Commented Sep 5 at 18:27
  • 2
    $\begingroup$ @dipetkov Fair point. Although to some extent this is always the case when somebody who doesn't know about certain relevant procedures does one thing they know, then writes here to ask about what they want, and then learns about other things. Also to me it doesn't look like they wanted a significant result from the first analysis already, rather they say "a non significant result makes sense", and it may well be that had they known about equivalence tests in advance, they had chosen them from the start. (Of course I realise I'm generous here.) $\endgroup$
    – Christian Hennig
    Commented Sep 5 at 19:29
9
$\begingroup$

I agree with Christian Hennig's advice on equivalence tests (+1).

In addition, I think it can be useful to do a power analysis for the minimum effect size that would be interesting/meaningful in your case. This would tell you if your study was appropriately powered to detect an interesting effect, and if not, how much more data would have been required. Note that to avoid bias, it is important that this is based on the minimum effect size of interest, and not the effect size that you actually found in your analysis.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I think this goes in the direction of what I was thinking. How would you choose the minimum effect size though? Would it be based on the theory or the data? $\endgroup$
    – PostDocing
    Commented Sep 5 at 10:57
  • 3
    $\begingroup$ @PostDocing Ideally based on theory. If that is not sufficiently developed or precise, I would based it on existing knowledge. $\endgroup$
    – mkt
    Commented Sep 5 at 12:03
  • 1
    $\begingroup$ @PostDocing Not based on the data, since that will just recapitulate the finding of the hypothesis test. Using the observed effect size, you'll always find you had high power if the p-value is significant, and low power if it's not. This is called post-hoc power analysis, and is easily misinterpreted as giving new information. $\endgroup$
    – Nuclear Hoagie
    Commented Sep 5 at 18:57
  • 2
    $\begingroup$ @PostDocing In addition to other comments, in general, if there are specific, identified costs and benefits associated with the effect size, you could use that to choose a minimum effect size. To take your example of an intervention increasing life expectancy, at which point do the benefits outweigh the side effects or costs of the intervention? It requires defining what is a cost or a benefit. Reduced quality of life could count as a cost. Would this intervention divert resources from a more cost-efficient intervention? etc. $\endgroup$
    – J-J-J
    Commented Sep 6 at 5:20
3
$\begingroup$

Also agreeing with Christian Hennig's advice on equivalence tests (+1).
Just pointing out here that you may not need to run any new test, compute any new p-value, or any such. You may very well already have the answer...
You say you are doing a regression analysis, and you obtained a point estimate which is not significant. Let's pretend this point estimate is a slope (if it is not, you can adjust my example, but the principle will remain). You should then also have a confidence interval (CI) for the slope; that CI contains 0 (because it was not significant).
Now, based on expert domain knowledge, you need to decide what slope would be considered practically significant; probably not .0001, but say, for example .5.
Then look at the CI; does it contain .5 (or contain either .5 or -.5, for a 2-sided test). If not, you have now ruled out a slope of .5 or greater, and thus can say that you did not miss a practically significant effect due to lack of power.

You asked "how do I quantify the precision of my estimate"; the width of your CI does that. $100(1-\alpha)$% of the time, the CI will contain the true value of the parameter; the width of the CI tells you how "tight" your estimate of its value is.

Improve this answer
$\endgroup$
5
  • $\begingroup$ You're not wrong, but I'd be leery about drawing strong conclusions without knowing much more about the data and study. Relying on the CIs / p-values alone can lead to some wildly exaggerated or incorrect inferences: statmodeling.stat.columbia.edu/2014/11/17/… & statmodeling.stat.columbia.edu/2023/09/07/… . $\endgroup$
    – mkt
    Commented Sep 6 at 9:15
  • 1
    $\begingroup$ @mkt, you are absolutely correct, for significant results with low power / small sample sizes. The OP has a non-significant result. But there could still be an issue with low power/sample size; which would be an issue with the p-value and CI (non-representative). I would love to see the same graph as your 2nd link, but for the, say, 1000 first non-significant CI's; how "wild" are they? Do they have the wrong sign? Are the CI's representative? $\endgroup$
    – jginestet
    Commented Sep 6 at 16:39
  • $\begingroup$ Yes, agreed. This is certainly a case where your suggestion would be less of an issue (maybe not an issue at all?). I mention the possible problem lest future readers apply this logic more generally, including to statistically significant results. $\endgroup$
    – mkt
    Commented Sep 6 at 16:48
  • $\begingroup$ @mkt The Gelman blog discusses what happens if you look only at the "significant" results. If you don't do any filtering, interpreting values outside CIs as "rejected" (with some confidence, assuming model is roughly correct, etc.) is IMHO pretty much OK. $\endgroup$
    – Martin Modrák
    Commented Sep 7 at 18:31
  • 1
    $\begingroup$ @MartinModrák I mostly agree (see my previous comment) but I think the lesson is a bit more general than that. There are multiple ways in which parameter estimates and uncertainties can be biased. The most obvious is selecting on significance. But other data-dependent decisions would also violate the assumptions behind a CI. I'm not saying that the OP has made any such error - just that we ought to be cautious about offering a caveat-free interpretation because (i) we don't know all the details here and (ii) future readers may miss this important point about significance/non-significance. $\endgroup$
    – mkt
    Commented Sep 7 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.