4
$\begingroup$

The model I have for the regression is in accordance with literature. I have about 300k observations and 25 dependent variables. When I run OLS regression, the output shows a couple of significant variables, which is satisfactory. However, the adjusted r-squared is very low: 0.000960!!! IS THIS ALARMING? Other estimators are:

S.E. of regression  0.027463
Sum squared resid   227.8050
Log likelihood          657295.4
F-statistic         11.61202
Prob(F-statistic)   0.000000
Mean dependent var  0.000960
S.D. dependent var  0.027475
Akaike info criterion  -4.351855
Schwarz criterion      -4.350941
Hannan-Quinn criter.   -4.351591
Durbin-Watson stat  1.271290

Depending on the information given above, what would be your evaluation of the model and the regression? Please advise me.

Much thanks!

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ This is not a strong model but because the sample size is so large you can detect a small positive R$^2$. $\endgroup$
    – Michael R. Chernick
    Commented May 24, 2019 at 3:17
  • $\begingroup$ I unfortunately cannot reduce the sample size; it has to be like this. Can the low adjusted r-squared owing to the large sample size cast any doubt on the significance of the variables? Much thanks for your response. $\endgroup$
    – Deepan Das
    Commented May 24, 2019 at 3:23
  • $\begingroup$ I wasn't suggesting reducing the sample size. The larger the sample size the better. A low R$*2$ could indicate a poor model. The fact that it is statistically significantly different from 0 is useful information. $\endgroup$
    – Michael R. Chernick
    Commented May 24, 2019 at 17:57
  • $\begingroup$ Adjusted $R^2$ is just one way to guard against overfitting. Have you tried to train on 250k samples and find $R^2$ for the predictions of the 50k that your model did not see during training? $\endgroup$
    – Dave
    Commented Jul 12, 2019 at 0:43

2 Answers 2

Reset to default
4
$\begingroup$

Welcome to CV!

An $R^2$ of $0.00096$ means that your regression model explains $0.096\%$ of the variance in the response variable. This is an extremely small amount, so it means your model does not explain the variance in the response variable well.

The reason you still obtain significant estimates for the coefficients is that your sample size is extremely large. Just think of what significance of a coefficient means: The estimate $\pm$ the standard error times some amount (depending on the level of significance) does not contain zero. If your sample size $n$ is very large, your standard errors will be almost $0$, because the standard error is $\frac{s}{\sqrt{n}}$. Significance therefore has little meaning with large $n$. Your effect sizes could be extremely small, but still significantly non-zero.

If anything, you can conclude that your 25 variables explain almost none of the variance in the response. Perhaps there is not a linear relationship between them and the outcome, or perhaps they are just not related to the outcome at all.

Improve this answer
$\endgroup$
0
3
$\begingroup$

The low adjusted r-squared suggests that your model is not accounting for much variance in the outcome. This means that the associations between your predictors and outcome are not very strong. However, with such a large sample you have enough statistical power to detect even small effects. This leads to cases like yours, where a model with a small r-squared has statistically significant effects (even if these effects may be too small to be considered meaningful). If you are concerned about whether the results of your model are consistent with the literature, you can compare the strength of the effects you're observing with those that have been reported in the past.

You can find more discussion about p-values and effect sizes here:

Cumming, G. (2014). The New Statistics: Why and How. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.