4
$\begingroup$

I have a conceptual question regarding linear regression.

Assume our model is correct, i.e., the response variable $Y$ is indeed coming from the model

$$Y=\beta_0+\beta X+\epsilon.$$

Here $X$ is a vector of length $m$. Assume all the nice assumptions hold, e.g., $\epsilon$ is normal, we have a set of i.i.d. observations. We know that by consistency of OLS estimator, we will be able to reach the true value of those coefficients (some of the coefficients could be 0) if we have an infinite amount of observations. My question is, given the fact that we only have a finite amount of data, is there a method that can correctly identify those non-zero coefficients at some confidence level?

According to "An Introduction to Statistical Learning" (and I agree with this book on page 77), we cannot simply look at the $p-$value associated with each individual coefficient and claim that if one coefficient has $p-$value less than 5%, then we conclude this coefficient is non-zero at 95% confidence. In this book, it says that this logic (looking at each individual $p-$value) is flawed, especially when we have large number of predictors. Because if we have 100 predictors, then about 5% of the $p-$values will be below 0.05 just by chance, even though it could be that the true model has all coefficients being 0. That's why we need to look at the $F-$test of the overall significance of the model, i.e., whether there is at least one coefficient that is significantly different from 0.

I fully agree with this argument. But then what is the point of those $p-$values of each individual coefficient?

And the next question is: if the $F-$test concludes that among all predictors, there is at least one coefficient that is not 0 at significance level $\alpha=0.05$, we don't know which coefficient is (coefficients are) significantly different from 0. We know that if $\text{H}_0:$ all coefficients are 0 is true, then there is only $\alpha$% of chance that the $F-$test will result in a $p-$value below 0.05, regardless of the number of predictors and number of observations. So now if $F-$test rejects $\text{H}_0$, in this case, if we find there are 12 coefficients whose $p-$values are below $\alpha$, can we conclude that these 12 coefficients are non-zero? At what confidence level? $1-\alpha$? Or something else? How to interpret the individual $p-$values (the $t-$tests) together with the overall $p-$value (the $F-$test)?

If we perform forward selection/ backward elimination/ step-wise selection, in the final set of predictors produced by these methods, at what confidence level can we conclude that those corresponding coefficients are non-zero?

If we run $2^m$ regression model (where $m$ is total number of predictors), will this help us to identify the set of non-zero coefficients at least theoretically? At what confidence level?

$\endgroup$
  • $\begingroup$ Maybe you should look at ''multiple comparison procedures'' like e.g. Bonferroni , Holm, ... $\endgroup$ – user83346 Feb 21 '16 at 11:39
  • $\begingroup$ Let me play the devil's advocate here: If the model is "correct" (correctly specified), then none of the coefficients is zero, almost by definition. And while it's a rule of thumb that coefficients not statistically significantly different from zero are removed, I'm reading more and more lately about this being the wrong thing to do. This is assuming a principled and correctly specified model. If you're just throwing a kitchen-sink-full of variables into a "model", things would be different: starting with the idea that the model is correctly specified. $\endgroup$ – Wayne Feb 21 '16 at 14:58
3
$\begingroup$

I do not have a good answer, but let me rephrase some of you thoughts and questions and give comments.

<...> what is the point of those $p$−values of each individual coefficient?

A $p$-value is a valid tool when assessing the significance of a single regressor individually. If you care about whether $X_i$ has a non-zero coefficient in population, you look at the $p$-value associated with the coefficient $\beta_i$ at $X_i$. If this is the only question you have, this is a satisfactory answer. That is the point of the individual $p$-values.

That's why we need to look at the $F$−test of the overall significance of the model, i.e., whether there is at least one coefficient that is significantly different from 0.

Yes, the $F$-statistic informs you whether all the regressors taken together have zero coefficients in population. But this is only a special case of what you are interested in, if I understand you correctly. So the $F$-statistic is not useful here -- unless it is low enough to conclude that at the given significance level there is not enough evidence to reject the null.

<...> if we find there are 12 coefficients whose $p$−values are below $\alpha$, can we conclude that these 12 coefficients are non-zero? At what confidence level? $1−\alpha$? or something else?

You can take any single coefficient individually and conclude at $1-\alpha$ confidence level that it is non-zero, but you cannot do that jointly for all 12 coefficients at $1-\alpha$ confidence level. If the significance tests for the twelve coefficients were independent, you could say that the confidence level is $(1-\alpha)^{12}$ which is (considerably) lower than $1-\alpha$.

If we perform forward selection/ backward elimination/ step-wise selection, in the final set of predictors produced by these methods, at what confidence level can we conclude that those corresponding coefficients are non-zero?

This is a tough question. The $p$-values and the $F$-statistic in the final model are conditional on how that model was built, i.e. on the forward selection / backward elimination / step-wise selection mechanism. Hence, they cannot be used as is for making inference about whether the coefficients are zero in population; these values have to be adjusted. There may exist a procedure for that (because the issue has been known for a long time), but I cannot remember any relevant reference.

If we run $2m$ regression model (where $m$ is total number of predictors), will this help us to identify the set of non-zero coefficients at least theoretically? At what confidence level?

Recall that all the models that will have omitted variables (variables that have non-zero coefficients in population) will suffer from the omitted variable bias and generally will have "wrong" $p$-values etc., so the approach appears problematic.

Finally, note that the "nice assumptions"

Assume all the nice assumptions hold, e.g., $\epsilon$ is normal, we have a set of i.i.d. observations.

require $\epsilon$ -- rather than the observations $Y$ and $X$ -- to be i.i.d.

$\endgroup$
  • $\begingroup$ I agree with you, all of your comments. So in practice, I think to do inference, e.g., confidence interval of those parameters, and run those statistical test, is somewhat less meaningful. Because we never know "quantitatively" how close our model is to the real data generating process. However, if our goal to run this regression model (not necessarily linear) for intervention rather than just pure prediction, then we have to do inference on those coefficients..... $\endgroup$ – KevinKim Feb 21 '16 at 15:47
  • $\begingroup$ As I said, I just partly rephrased your post, so I am not surprised that we agree :) Since Frank Harrell has answered the post, too, I think you may benefit from familiarizing youself with his approach. He must have lots of experience, although perhaps (I am not sure) in a specific setting. $\endgroup$ – Richard Hardy Feb 21 '16 at 15:51
  • $\begingroup$ In my question "run $2^m$ regression models...." I think even if all of those predictors are uncorrelated, in which case, we don't have an omitted variable bias, I think still we cannot claim which set of coefficients are non-zero at a specified confidence interval. $\endgroup$ – KevinKim Feb 21 '16 at 15:59
  • $\begingroup$ I think the reason is because, when we dump some predictors into the error term, the distribution of the "new error term" is no longer normal in general, because it becomes a combination of normal (which is the original error term by assumption) and the distribution of the predictors that have been dumped into the error term. Hence, we don't know what is the distribution of the new error term, therefore, we cannot run any inference $\endgroup$ – KevinKim Feb 21 '16 at 16:03
  • $\begingroup$ ...unless the sample is large enough so that we can rely on the central limit theorem and treat the estimated coefficients as coming from a normal distribution. $\endgroup$ – Richard Hardy Feb 21 '16 at 17:21
3
$\begingroup$

Running statistical algorithms to find the "correct" variables is almost futile. A relevant simulation is in Section 4.3 of my Regression Modeling Strategies course notes at http://biostat.mc.vanderbilt.edu/RmS#Materials

The simplest way to look at the difficulty of the task is to use the bootstrap to get confidence intervals for the importance rankings of variables competing in a multivariable regression. A demonstration of this is in Section 5.4 of the same course notes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.