5
$\begingroup$

I am writing a routine for logistic regression with lasso in matlab. So the problem is to minimize the negative log-likelihood function with the penalty term

$$\sum \left(\log(1 + e^{X_i' \beta}) - y_i X_i' \beta\right) + \lambda \sum |\beta_i|$$

where $\beta$ is the model parameter, $X_i$ is the $i$th row of matrix $X$, and $y_i$ is the value of observation $i$.

My first question is for a 5-fold cross-validation, which criterion should I use to pick the best value of $\lambda$? Should I use the value of the logit function on the validating data set or mis-classification rate on the validating data?

$\endgroup$
3
$\begingroup$

The short answer is, its up to you, depending on your interest. In the past I have used AIC for Lasso.

However it sounds like you are using this model for prediction, and thus using the mis-classification rate is a good idea. However misclassification can be categorized in many ways. Are you interested in the the absolute % classified correctly? Or maybe you just care about of those classified as 1 (or yes, etc), how many of those were classified correctly? I would do some reading into Positive Predictive values, Negative predictive values, etc.

https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values

In addition when doing your cross validation, there are a plethora of criteria you could use to validate your model. A short list of other common criterion are:

  • $R^2$
  • $MSE$
  • $Mallow’s$ $C_p$
  • $AIC$

Look them up and see which is most relevant to you!

$\endgroup$
  • 1
    $\begingroup$ How would you use the AIC to optimize the lasso? If each of your models is only different because it has a different value for $\lambda$, don't all the your possible models have the same number of free parameters? $\endgroup$ – Louis Cialdella Feb 2 '16 at 2:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.