2
$\begingroup$

How much can we trust the "statistical significance" (with a p-value inferior to 0.05 for example) of the t-test in Linear Regression? Are there cases where we have a statistical significant variable, but still the variable is useless? In scientific papers, I often see authors creating variables that try to somehow bruteforce statistical significance, they try different variables until they find one that is statistically significant and they conclude here.

$\endgroup$

1 Answer 1

2
$\begingroup$

There are probably countless articles and posts and papers about how the meaning of 'statistical significance' can range from meaning a lot to meaning nothing, for three posts on Cross Validated, see here or here or here.

A careful reading of these should clear up most of your questions, and so I'll instead focus on two points that are echoed throughout. First, the textbook definition of a p-value and statistical significance implies that observing a pvalue less than the (a priori) determined level means you reject your null hypothesis, and observing a higher pvalue means you fail to reject your null hypothesis. In a linear regression, rejecting the null means you reject the hypothesis that the coefficient is zero, given the assumptions you made to obtain your estimate and your test statistic. That last part is important, because it includes lots of obvious caveats, and even more subtle ones. For example, on obvious one is that your model could be incorrectly specified... in a linear regression, what if the true relationship is non-linear? What if the error terms are not normally distributed, and your given your limited number of observations, the asymptotic normality fails too? Then your test is assuming things that simply don't hold in your model, and so it has to be incorrect. How incorrect? Well it depends on your test. A little more subtle is the fact that just looking at the final pvalue reported may fail to account for multiple hypothesis testing... simply running tons of tests and choosing the one that gives you a low pvalue has a very different interpretation than running one test and not changing anything after seeing the value. As you mention, in published work, it's extremely hard to know how much mucking around happened beforehand to produce these results, and that's one important thing peer-reviewing attempts (but could be argued often fails) to keep in check (there's loads of work about publication bias that may interest you).

So the short answer is you can trust 'statistical significance' as much as you trust the entire research process, and that underlies an important point that the pvalue on it's own, though it can be a good summary of your results conditional on good research practices and careful thought about your model and testing, should never be considered alone.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.