2
$\begingroup$

I am estimating an OLS regression with 158 observations and about 140 regressors (some unstructured data features to personality measure). Below is the bottom output I get from OLS. Some of the regressors turn out to be significant at 0.05 level. But I am wondering why may there be a case when adjusted R-squared is reasonable but p-value for F-test is not. Does this mean that the model is not valid even though some of the regressors are significant and adjusted R-square is reasonable?

And what could be the reason causing this and how to remedy this?

Residual standard error: 0.5457 on 19 degrees of freedom
Multiple R-squared: 0.9083, Adjusted R-squared: 0.2424
F-statistic: 1.364 on 138 and 19 DF, p-value: 0.2205

Improve this question
$\endgroup$
3
  • 6
    $\begingroup$ Your results make a lot of sense as you're dramatically overfitting to your data. You basically have the same number of observations as regressors. You need to dramatically reduce the number of regressors you're using (~10). This shows the pitfalls of the R-squared. Your model is bad, but the R-squared is still really high. $\endgroup$
    – Jacob H
    Mar 26, 2019 at 6:20
  • 1
    $\begingroup$ Possible duplicate of High R-squared although many insignificant coefficients $\endgroup$
    – kjetil b halvorsen ♦
    Mar 26, 2019 at 9:34
  • $\begingroup$ I don't think this is a duplicate. In that problem, there were lots of missing data and the overfitting (even with listwise deletion) was not nearly so severe. $\endgroup$
    – Peter Flom
    Mar 26, 2019 at 11:03

1 Answer 1

Reset to default
4
$\begingroup$

Don't just look at adjusted $R^2$, look also at the gap between that and the regular $R^2$. Here it is huge (dropping by 0.65). That is a sign of overfitting.

Also, you have almost as many regressors as observations. That usually leads to overfitting.

You could make up random noise and get results like this.

You almost certainly also have collinearity.

It's hard to believe that any study of personality needs or could use 140 variables.

So, what to do? There are many options for data reduction in a case like this. One that I like is partial least squares regression. This is similar to principal components regression, but also accounts for relationships with the DV. However, if your goal is to look at personality in terms of latent factors (a common thing to want to do) then a factor analysis of the 140 IVs may be better, and you could then use the factors as IVs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.