I have read in several places that R Squared is not an ideal measure when a model is fit using LASSO. However, I'm not clear on exactly why that is.
In addition, could you recommend the best alternative?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
The goal of using LASSO is obtaining a sparse representation (of a predicted quantity) in the sense of not having many covariates. Comparing models with $R^2$ tends to favor models with lots of covariates: in fact, adding covariates unrelated to the outcome will never decrease $R^2$ and almost always increases it at least a little bit. The LASSO model will identify the model with the optimal penalized log-likelihood (an unpenalized log-likelihood is monotonically related to the $R^2$). Validation statistics that are more widely used to compare LASSO models to other types of models are, for instance, the BIC or cross-validated $R^2$.
-
1$\begingroup$ +1 for clearly present the reason and provide alternative $\endgroup$ Commented Jun 8, 2018 at 17:50
-
1$\begingroup$ Thanks very much for the great answer! Would you mind elaborating on "The LASSO model will identify the model with the optimal penalized log-likelihood (an unpenalized log-likelihood is monotonically related to the R2)." I take the first part to mean that it will choose the model with the least amount of error (in prediction and via the penalization)? But I'm unclear on what the bit in brackets means. Does that mean that unpenalized LL goes up as R2 goes down? Also, does the cross-validated R2 have to be in an entirely new dataset? Or can it be based on the training data? $\endgroup$– DaveCommented Jun 8, 2018 at 18:10
-
3$\begingroup$ @Dave I think you have the right idea. The linear regression model is a LASSO with no penalty, and the log-likelihood is just $\log(2\pi)N+1−\log(N)+\log(\sum_{i=1}^n r_i^2)$ whereas the R2 is just $1 - \sum_{i=1}^n r_i^2/\sum_{i=1}^ny_i^2$ . The penalization contributes to error indirectly, it is a price you pay to enforce sparseness. The unpenalized model will always have lower (internal) error. People generally do cross-validation with the same dataset. Testing models in new datasets is a whole other thing(no need for the "cross" part) and it isn't done enough. $\endgroup$– AdamOCommented Jun 8, 2018 at 18:24
-
$\begingroup$ @AdamO I think it'd be a good idea to edit your comment into your answer, it's very good. $\endgroup$ Commented Jun 8, 2018 at 20:04
-
1$\begingroup$ Hi @AdamO one final follow up question. I understand now why traditional R2 is a bad measure. But, I'm unclear as to why cross-validated R2 (within the same dataset) is okay? $\endgroup$– DaveCommented Jun 8, 2018 at 21:27