5
$\begingroup$

I have built some "regular" and robust regression models, using the standard lm function as well as rlm and lmrob. While I know that there is some discussion about using stepwise regression, I have used the stepAIC function to prune my variable set. After I've gotten a reduced set of variables using stepAIC, I've then run some robust regressions.

The cvTools package allows me to use cross validation to compare the performance of my various models.

I obviously would like to run lasso regression (ideally using the glmnet package) on my dataset.

My question is whether or not there is an already built package/functionality that will allow me to use cross validiation to compare the lasso regression model with the other models.

If there is not, then my initial thought had been to go back to first principles and manually code K-fold cross validation for lasso regression. However, I am now wondering if this is theoretically a good idea. Each time I run a fold in my manual CV, I would run cv.glmnet on the training set. Each training set would most likely result in a different lambda.min and lambda.1se.

My question is: is it technically proper CV to determine the overall CV error by averaging the error on each fold given that the lambda chosen for each fold will be producing a different lasso result?

Here is some sample code that I have used to create leave-one-out CV on the dataset to evaluate the lasso regression. I have computed my cross validation error on each fold using lambda.1se and labmda.min that arise for that fold.

lassocv<-function() {

len<-length(drx$DR)

errmin<-0
err1se<-0
print(len)

for (i in 1:len) {
    gmer<-data.matrix(drx[-i,])
    yxer<-yx[-i]
    lfit<-cv.glmnet(gmer, yxer)
    newr<-data.matrix(drx[i,])
    pmin<-predict(lfit,newx=newr,s=lfit$lambda.min)[[1]]
	    p1se<-predict(lfit,newx=newr,s=lfit$lambda.1se)[[1]]
    errmin<-errmin+abs(pmin-yx[i])
    err1se<-err1se+abs(p1se-yx[i])
}
print(errmin/len)
print(err1se/len)

}

However, I get different CV results. The two results that are returned for my dataset are 21.94867 and 23.74074.

$\endgroup$

migrated from stackoverflow.com Sep 26 '15 at 8:58

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Maybe look at something like this with the caret packge? machinelearningmastery.com/… Not sure if your request needs to use cvTools, but caret will do the cross validation for you as well. $\endgroup$ – Steve_Corrin Sep 25 '15 at 21:55
1
$\begingroup$

I'm not entirely sure I understand precisely where in the analysis pipeline your question is, but I think I can address it by walking through the steps you'll want to take. The software portion of your question is off-topic on CV, but the questions about CV are on-topic, so I'll answer those.

My question is: is it technically proper CV to determine the overall CV error by averaging the error on each fold given that the lambda chosen for each fold will be producing a different lasso result?

The elementary model development process is usually presented with respect to three partitions of your whole data set: train, test and validate. Training and test data are used together to tune model hyperparameters. Validation data is used to assess the performance of alternative models against data that wasn't used in model construction. The notion is that this is representative of new data that the model might encounter.

A slightly more sophisticated elaboration on this process is nested cross-validation. This is preferred because, across the whole process, all data is eventually used in testing and training the model. Instead of using one partitioning of the data, you can do CV partitioning on the whole data set (the outer partition) and then again on the data left over when you hold out one of the outer partitions (the inner set). Here, you tune model hyperparameters on the inner set and have out-of-sample performance evaluated on the outer holdout set. The final model is prepared by composing a final partition over the entire data set, using CV to select a final tuple of hyperparameters and then, at last, estimating a single model on all available data given that selected tuple. In this way, the model building process kind of telescopes on itself, collapsing CV steps as we estimate the final model.

It doesn't matter that alternative inner sets might give you different $\lambda_\text{min}$. What you're characterizing with your out-of-sample performance metrics is the model selection process itself. At the end of the day, you'll still only estimate one model, and that's the value of $\lambda_\text{min}$ that you care about. In the preceeding steps, you don't need to know the particular value of $\lambda_\text{min}$ except as a means to achieve out-of-sample estimates.

While I know that there is some discussion about using stepwise regression, I have used the stepAIC function to prune my variable set.

This is a bit of an understatement: it's not a discussion, it's a consensus that stepwise results are dubious. If you're fitting a lasso anyway, you can get statistically valid model by omitting the stepwise regression step from your analysis. Moreover, since the lasso step won't "see" the stepwise step, your results will have too-narrow error bands and cross-validation results will be irreparably biased. And lasso makes the entire stepwise step pointless anyway, because they solve the same problem! Lasso solves all of the variable selection problems that stepwise attempts to while avoiding the wealth of widely-accepted criticisms of stepwise strategies. There's no downside to using lasso on its own in this case. I'm convinced the only reason stepwise methods are included in R is for pedagogical reasons, and so that the functionality is available should someone need to demonstrate why it's hazardous.

$\endgroup$
  • $\begingroup$ Your point that the model selection process is what's being characterized is too often missed, glad you emphasized that. In fairness to stepwise selection, even Frank Harrell's rms package allows for backward selection, which can be helpful in some restricted circumstances. He of course goes to pains in the package manual and in the accompanying book to warn of the dangers. $\endgroup$ – EdM Sep 26 '15 at 17:48
  • $\begingroup$ Given that this analysis includes lasso, is there any advantage to be found in also performing stepwise? I'm genuinely asking -- it seems like a hat on a hat. $\endgroup$ – Sycorax Sep 26 '15 at 17:59
  • $\begingroup$ With LASSO I agree that there is no obvious need for prior variable selection, particularly stepwise. My point was just that there can be circumstances where stepwise can be used. Frank Harrell does say: "Do limited backwards step-down variable selection if parsimony is more important than accuracy" as a late step in model building, on page 97 of Regression Modeling Strategies, second edition. So there is a reason other than pedagogical to keep stepwise methods in R. $\endgroup$ – EdM Sep 26 '15 at 22:02
  • $\begingroup$ Thank you for your resopnse. However, maybe I'm not following your answer, but when I try to write leave-one-out CV, I get a different overall CV error depending on what lambda value I chose on each fold. $\endgroup$ – Jonathan Sep 27 '15 at 20:26
  • $\begingroup$ @Jonathan Looking at the code you posted, it seems that you should expect to get two different answers, one for lambda.min and one for lambda.1se... But I don't understand how the code snippet bears on your question, since my understanding is that you're asking whether it's OK to average out-of-sample errors for different lambda values (the answer is yes), and no part of the code snippet establishes that lambda values are different. $\endgroup$ – Sycorax Sep 27 '15 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.