2
$\begingroup$

The model I have for the regression is in accordance with literature. I have about 300k observations and 25 dependent variables. When I run OLS regression, the output shows a couple of significant variables, which is satisfactory. However, the adjusted r-squared is very low: 0.000960!!! IS THIS ALARMING? Other estimators are:

S.E. of regression 0.027463 Sum squared resid 227.8050 Log likelihood 657295.4 F-statistic 11.61202 Prob(F-statistic) 0.000000 Mean dependent var 0.000960 S.D. dependent var 0.027475 Akaike info criterion -4.351855 Schwarz criterion -4.350941 Hannan-Quinn criter. -4.351591 Durbin-Watson stat 1.271290

Depending on the information given above, what would be your evaluation of the model and the regression? Please advise me.

Much thanks!

$\endgroup$
  • $\begingroup$ This is not a strong model but because the sample size is so large you can detect a small positive R$^2$. $\endgroup$ – Michael Chernick May 24 at 3:17
  • $\begingroup$ I unfortunately cannot reduce the sample size; it has to be like this. Can the low adjusted r-squared owing to the large sample size cast any doubt on the significance of the variables? Much thanks for your response. $\endgroup$ – Deepan Das May 24 at 3:23
  • $\begingroup$ I wasn't suggesting reducing the sample size. The larger the sample size the better. A low R$*2$ could indicate a poor model. The fact that it is statistically significantly different from 0 is useful information. $\endgroup$ – Michael Chernick May 24 at 17:57
  • $\begingroup$ Adjusted $R^2$ is just one way to guard against overfitting. Have you tried to train on 250k samples and find $R^2$ for the predictions of the 50k that your model did not see during training? $\endgroup$ – Dave Jul 12 at 0:43
1
$\begingroup$

Welcome to CV!

An $R^2$ of $0.00096$ means that your regression model explains $0.096\%$ of the variance in the response variable. This is an extremely small amount, so it means your model does not explain the variance in the response variable well.

The reason you still obtain significant estimates for the coefficients is that your sample size is extremely large. Just think of what significance of a coefficient means: The estimate $\pm$ the standard error times some amount (depending on the level of significance) does not contain zero. If your sample size $n$ is very large, your standard errors will be almost $0$, because the standard error is $\frac{s}{\sqrt{n}}$. Significance therefore has little meaning with large $n$. Your effect sizes could be extremely small, but still significantly non-zero.

If anything, you can conclude that your 25 variables explain almost none of the variance in the response. Perhaps there is not a linear relationship between them and the outcome, or perhaps they are just not related to the outcome at all.

$\endgroup$
  • 1
    $\begingroup$ Thanks for enlightening. $\endgroup$ – Deepan Das May 24 at 3:40
1
$\begingroup$

The low adjusted r-squared suggests that your model is not accounting for much variance in the outcome. This means that the associations between your predictors and outcome are not very strong. However, with such a large sample you have enough statistical power to detect even small effects. This leads to cases like yours, where a model with a small r-squared has statistically significant effects (even if these effects may be too small to be considered meaningful). If you are concerned about whether the results of your model are consistent with the literature, you can compare the strength of the effects you're observing with those that have been reported in the past.

You can find more discussion about p-values and effect sizes here:

Cumming, G. (2014). The New Statistics: Why and How. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.