Suppose you add a feature $x$ to an OLS model. The AIC goes down by 100 but the new feature is not statistically significant. The expanded model should be preferred overall but is the estimated coefficient on $x$ any good? According to statistical theory if it's not significant you can't reject the possibility that it's $0$.
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ "According to statistical theory if it's not significant it's likely 0." This is not what a p-value tells you. $\endgroup$– DaveCommented Aug 5, 2020 at 22:31
-
$\begingroup$ I meant colloquially. I clarified it now. $\endgroup$– badmaxCommented Aug 5, 2020 at 22:56
-
1$\begingroup$ It will be helping if summary output is inserted in your question. $\endgroup$– user10619Commented Aug 6, 2020 at 5:58
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
According to statistical theory if it's not significant it's likely 0.
That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.
Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.
Third of all, including or excluding a variable has little to do with significance as I explain here.
EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."
And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?
The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither am I), and I imagine you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?
answered Aug 5, 2020 at 22:57
-
$\begingroup$ I would like to estimate the effect size associated with this new variable. A statement like "by increasing the value of $x$ by 1 unit, $y$ goes up by $n$ units'. In regression $n$ would be the estimated coefficient $\beta$, but is still still valid under the conditions I describe? $\endgroup$– badmaxCommented Aug 5, 2020 at 23:27
-
$\begingroup$ @badmax absolutely. You can’t reject the null in the typical sense, but hypothesis testing has no effect on estimation. You’re free to make the estimates you mention. I would also construct a confidence interval and report that. $\endgroup$ Commented Aug 5, 2020 at 23:49
-
1$\begingroup$ I do wonder what we should make of a point estimate of $5$ but a confidence interval of $(-5, 15)$, though, especially if a value like $5$ would have practical significance. $\endgroup$– DaveCommented Aug 6, 2020 at 1:49
-
1$\begingroup$ @Dave Frank Harrell and company have an interesting thread on this here. In your example, if the interval covered practically significant values then that would mean that practically significant values are consistent with the data. Additionally, a salient argument against assuming the effect is null is the classic "absence of evidence is not the absence of evidence", but this may have its own problems as has been argued here. $\endgroup$ Commented Aug 6, 2020 at 3:19
-
1$\begingroup$ "...when the p value is greater than 0.05 (let's call it p), then there is a p probability that we observe effects ..." Not to nitpick, but isn't this statement true no matter what value p has? $\endgroup$– nopeCommented Aug 6, 2020 at 4:49