0
$\begingroup$

When I run the linear regression using about 12 independent variables, I get insignificant F-test result overall. So I discarded variables to make the F-test significant while having no multicolinearity problem checking by VIF test.

Then I come up with a linear regression with significant F-test and no multicolinearity problem. However, I am left only with three independent variables and significant t-test results for these coefficients.

Shall I still use this result?

Improve this question
$\endgroup$
11
  • 3
    $\begingroup$ I performed an analysis exactly like yours, using $120$ observations and $12$ independent variables, on $1000$ independent datasets. In $47\%$ of those cases, I got an insignificant $F$ statistic (at the $95\%$ level) but found three variables that gave a significant $F$ statistic. What ought to interest you about this work is that the response variable was generated purely at random, independent of the explanatory variables. This suggests your procedure has a high chance of identifying a "significant" model when in fact there is no relationship whatsoever among any of the variables. $\endgroup$ – whuber♦ Apr 13 '17 at 16:13
  • $\begingroup$ So do you recommend using (or not using) this result? Actually, these variables are theoretically relevant to the dependent variable although I find only 3 of them to be significant with no muticolinearity and significant overall F-test. $\endgroup$ – Eric Apr 13 '17 at 16:16
  • 3
    $\begingroup$ I am not making a recommendation, because I do not know all the details of your situation: but I have provided good evidence that (to the extent your problem describes the essential features of your analysis) you run a high risk of being wrong if you do rely on your result. Moreover, the procedure of discarding variables until you find significance has a good chance of removing the wrong variables. There are much better ways: search this site for posts on model identification and selection. $\endgroup$ – whuber♦ Apr 13 '17 at 16:21
  • 1
    $\begingroup$ That's one set of tools you might consider. Entire books have been written on this subject, such as Frank Harrell's Regression Modeling Strategies. All multiple regression textbooks discuss various variable-selection and -checking procedures and provide advice on when to consider them (although not all of them work equally well!) $\endgroup$ – whuber♦ Apr 13 '17 at 16:26
  • 1
    $\begingroup$ Your ridge and Lasso results appear to confirm the spuriousness of your original conclusions. When an ad hoc procedure, with no theoretical justification, gives answers that conflict with well-established applicable procedures, it is unwise to trust the results of the ad hoc procedure. $\endgroup$ – whuber♦ Apr 13 '17 at 19:20
3
$\begingroup$

For variable selection try using LASSO or ridge regression. Both of these perform variable selection. LASSO has the added benefit of zero out coefficients of insignificant variables.

Both are forms of penalized regression. The penalization parameter can be obtained with cross validation.

All of this can be done with R, using the glmnet package and the glmer() and cv.glmer() functions.

Another approach is use a validation set to compare error rates from models or use area under ROC curves. It really depends on what you are trying to do.

As far as number of variables, as long as OLS assumptions are met, yes.

Improve this answer
$\endgroup$
8
  • $\begingroup$ I currently do not suffer any multicolinearity problem and the F-test and T-test are both significant with three independent variables. May I know the reason why using this result may not be ok first? I am using cross sectional data in this case. $\endgroup$ – Eric Apr 13 '17 at 15:50
  • $\begingroup$ Check model assumptions:en.wikipedia.org/wiki/… Typically it is pretty standard to check residual plots for heteroskedasticity and a qq plot of the residual for normality. R does this automatically with lm(). $\endgroup$ – Redeyes10 Apr 13 '17 at 15:56
  • $\begingroup$ Thanks but I am only worried about omitted variable bias. So I included 12 variables. But since there were problems with muticolinearity and F-test I discarded many only left with 3 independent variables. All I am trying to check is whether I can keep this result and report it. $\endgroup$ – Eric Apr 13 '17 at 16:00
  • $\begingroup$ Model assumptions must be satisfied. If you have area knowledge and justify dropping variables then it may be okay to report. This is why ridge and lasso were suggested for variable selection. $\endgroup$ – Redeyes10 Apr 13 '17 at 16:01
  • 1
    $\begingroup$ A very accessible (not math heavy) overview along with coding examples can be found in this text: www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf Overview: p 214 - 227 How to use glmer() p 251-255 $\endgroup$ – Redeyes10 Apr 13 '17 at 16:52
1
$\begingroup$

As whuber said, it's not really possible to recommend without more info. As already mentioned, make sure you really understand the assumptions ols regression makes, understand how to check such assumptions, and then you will be able to make the decision--and back it up. There are entire books on the topic as well as many online resources. Weissberg's Applied Linear Regression is fairly accessable start.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.