How is the R-squared calculated for an elastic net? How about LASSO? Should be different from OLS, or not?

Edit: The main problem is as follows:

We have all kinds of fruits like $f_1, f_2, ..., fn$ for which we have $5$ different properties $c_1,...,c_5$.

We know well 10 of the fruits which are $f_1,...,f_{10}$. For each of these known fruits, we want to find which of the remaining fruits in the set $f_i, 10<i<=n$ can better explain its features.

$f_j = \beta_if_i + \epsilon; 1<=j<=10; 10<i<=n$

To this end, I used elastic net due to the property of my data.

I used cross validation to fit my models and at the end I have 10 models each for one fruit $f_j; 1<=j<=10$.

For me, this is very important to see which of these models are very well fitted to the data. Then I can rank the models from $1$ to $10$ and use this ranking in further analysis.

In my question, fruit $f_k; 1<=j<=10$ might not be well explained by any other fruits! that's why I'm looking for a statistics that I can compare the goodness of fit for each of the obtained model.

I want now to know, whether or not the PRESS or R-squared is a good measure?

Thank you very much. N.


Generally speaking, if you are using a regularized method you typically have more variables than you have samples ($n\ll p$). As such, using R-squared is not recommended. This is because a model with all the variables will always have the largest R-squared. It is recommended to do some form of cross-validation.

If your dependent variable is continuous, you likely would use the Predicted Residual Sum of Squares (PRESS) statistic:

$$PRESS=\sum_{i=1}^{N} (y_{i}-\hat{y}_{i,-i})^2$$

If you are really attached the idea of an 'R-squared' you can convert the PRESS into R2 by dividing it by the sum of squares (SS) and subtracting everything from 1:

$$SS=\sum_{i=1}^{N} (y_{i}-\bar{y})^2$$ $$R2=1-PRESS/SS$$

If your dependent variable is a classifier (e.g. Control vs. Disease), then you would use a different metric like Accuracy, Area Under the Receiver Operator Curve (AUROC), or Kappa.

| cite | improve this answer | |
  • $\begingroup$ Thanks a lot for your clear response. I have a stupid question, what do you mean by \^{y}_{i,−i}. I don't get (i,-i), would you mind please shortly comment on it. $\endgroup$ – sbmm Nov 12 '14 at 8:32
  • $\begingroup$ @sbmm, it is simply the notation for the formal definition of PRESS. It signifies that $\hat{y}_{i}$ (i.e. the current sample) is predicted from from the remaining samples minus the current sample, hence the ${-i}$ $\endgroup$ – cdeterman Nov 12 '14 at 13:07
  • $\begingroup$ Imagine, I have an apple in one side and other fruits (except an apple) in the other side. I used elastic net to see which fruits, can better explain the features of the apple. I have 5 features for my fruits. Imagine my y(apple)=(1,1,2,1,3) and matrix X with size 20*5 which means 20 fruits with 5 features. I found a model per fruit via elastic net, and I found out that for e.g., the apple can be explained by pear and quince. Next, I take another fruit and continue the same. I want to see which of the fruits are better modeled by other. Would PRESS be a good statistics to compare my models? $\endgroup$ – sbmm Nov 12 '14 at 14:37
  • $\begingroup$ If I want to calculate the goodness of the fit for each fruit, which statistics should I apply? I I know it, then I can compare my models obtained for each fruit. $\endgroup$ – sbmm Nov 12 '14 at 14:41
  • $\begingroup$ Have you done some analysis? Do you have a dataset? I would likely be better able to help you if I can 'see' the dataset and how you have done your analysis. From what I understand, you are wanting to compare between classes (i.e. fruits) instead of classifying them from their respective features (which is what elastic-net and LASSO are for). In which case, you should be using a different method. You may wish to start another question to more directly address this. $\endgroup$ – cdeterman Nov 12 '14 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.