5
$\begingroup$

So I have a really small sample size of 50, and I have 80 regressors. The $R^2$ score is about 0.1, and according to the following equation on Wikipedia about how to compute adjusted $\bar{R}^2$,

$$ \bar{R}^2 = R^2 - (1-R^2)\frac{p}{n-p-1} \\ R^2 = 0.1 \\ p = 80 \\ n = 50 $$

Then the adjusted $\bar{R}^2$ shoots over to 2.42. But wikipedia says $\bar{R}^2$ should always be less than or equal to $R^2$, so what am I doing wrong here? or is it just the model is wrong since so many regressors?

Edit

Both $R^2$ and $\bar{R}^2$ were computed from lasso regression instead ordinary least squares.

Improve this question
$\endgroup$
10
  • $\begingroup$ how did you get the estimates of the regression coefficients? $\endgroup$
    – utobi
    Commented Nov 16, 2016 at 21:50
  • $\begingroup$ @utobi How is regression coefficients relevant? $\endgroup$
    – longtengaa
    Commented Nov 17, 2016 at 2:12
  • $\begingroup$ How did you get the fitted values? For $n<p$ you have an ill-posed problem and $(X^tX)^{-1}$ is not uniquely defined, so you can't compute $R^2$. Is that clear? $\endgroup$
    – utobi
    Commented Nov 17, 2016 at 14:19
  • $\begingroup$ @utobi yeah, you can't get a closed-form solution, but you can use optimization algorithms to approach a local minimum. Still, I don't see the relevance here.. $\endgroup$
    – longtengaa
    Commented Nov 17, 2016 at 19:14
  • 2
    $\begingroup$ Although $R^2$ is always meaningful as a relative decrease in variance of the response, for the adjustment to be useful there has to be a clear understanding of how many "degrees of freedom" are in the model. That's already questionable in the Lasso and, unless the regressors have sufficient linear dependence, is meaningless when there are more regressors than data. It won't tell you anything concerning whether the model is "wrong." $\endgroup$
    – whuber ♦
    Commented Nov 17, 2016 at 20:23

3 Answers 3

Reset to default
5
$\begingroup$

The adjusted $R^2$ value is specifically for linear regression where it's easy to know the effect of adding many predictors. If you were doing linear regression with more predictors than samples a linear regression would give $R^2=1$ so you must not be using a linear regression model. That means you can't adjust the $R^2$ figure regardless of how large your sample size is.

If you tried to adjust the $R^2$ value with your figures you'll notice that you get a value greater than $1$ and this is statistically meaningless.

But your question is still relevant if you had used linear regression with more predictors than samples and got $R^2=1$. You'll notice that when $p=n-1$ the adjusted $R^2$ is undefined, and in fact the adjustment isn't valid when $p\geq n-1$

Improve this answer
$\endgroup$
4
  • $\begingroup$ If by linear regression, you mean ordinary least squares (OLS), then I'm not using linear regression. The one I'm using has a L-1 regularization term (Lasso regression). So I have to use OLS to make $R^2$ valid? $\endgroup$
    – longtengaa
    Commented Nov 17, 2016 at 2:09
  • $\begingroup$ $R^2$ is valid for any model but adjusted $R^2$ is only valid for OLS. The adjustment won't work for lasso regression $\endgroup$
    – Hugh
    Commented Nov 17, 2016 at 9:36
  • $\begingroup$ can you elaborate more on why adjusted $R^2$ is only valid for OLS? $\endgroup$
    – longtengaa
    Commented Nov 17, 2016 at 19:15
  • $\begingroup$ @longtengaa OLS is mathematically simple, we know that adding more predictors affects the $R^2$ value in a specific way so we can adjust it with that formula to undo those specific effects. In any other model it's too complicated and we don't know how adding more parameters is going to change the $R^2$ value. Therefore we can't derive a formula for adjusting $R^2$ for the effects of many parameters. $\endgroup$
    – Hugh
    Commented Nov 17, 2016 at 19:22
2
$\begingroup$

To state notation, let $y$ be the $n$-vector of responses, let $X$ be the ($n\times p$) design matrix and let $\beta$ be the $p$-vector of unknown regression coefficients, with $n$ being the sample size. The well known least squares estimate of $\beta$ is $\hat\beta = (X^TX)^{-1} X^Ty$.

The coefficient of determination is $R^2 = 1-\frac{SS_{res}}{SS_{tot}}$, where $SS_{tot}$ is the total sum of squares and $SS_{res}$ is the residuals sum of squares. The adjusted $R^2$ is as you wrote.

Coming to you question, when $n<p$, $\hat\beta$ is not anymore uniquely defined because the inverse of $X^TX$ is not defined. Hence, as far as $n<p$, no matter what algorithm you use to find $\hat\beta$, the latter will always be undefined and arbitrary. Essentially, in this case, the objective function of $\beta$ is a flat surface. Consequently, $R^2$ is also arbitrary and therefore meaningless. For this reason, adjusted $R^2$ will be meaningless as well. That's why you obtain such a strange value for the adjusted $R^2$.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ @longtengaa I now saw your EDIT now. I agree with whuber's comment as far as lasso is concerned. $\endgroup$
    – utobi
    Commented Nov 17, 2016 at 20:26
  • 1
    $\begingroup$ Even when $X^\prime X$ fails to be invertible, $R^2$ nevertheless is defined. For insight, consider the extreme case where $X^\prime X$ is a zero matrix. Although $\hat\beta$ could be anything, $X\hat\beta$ is zero, implying all predicted values are zero. $R^2$ would then be the second moment of $y$ divided by the variance of $y$. $\endgroup$
    – whuber ♦
    Commented Nov 17, 2016 at 20:47
  • $\begingroup$ That's right. Actually, in the response I have in mind non trivial cases in which columns of X have at least one non-zero element. $\endgroup$
    – utobi
    Commented Nov 17, 2016 at 21:08
1
$\begingroup$

The adjusted R-square value is always less than R-square when n>p that means number of observation is greater than the number of parameters.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.