Return to Answer

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."

And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?

The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither am I), and I imagein you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."

And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?

The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither am I), and I imagine you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."

And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?

The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither and I), and I imagein you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."

And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?

The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither am I), and I imagein you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large of not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."

And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?

The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither and I), and I imagein you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?