I fitted a lasso logistic regression using glmnet. I use a pretty small dataset with only 51 (28/23) observations. I want to compare the model fit of two possible variable combinations.

  1. Only control variables
  2. Control variables + linguistic predictors

Both models are comparable regarding explained deviance with best lambdas (1.:17% | 2.:16% dev. explained from null model).

Now I want also compare the mean cross validated error at the best lambdas. Again both models are pretty close (1.: 1.304177 | 2.: 1.324639).

My questions are:

1.) What exactly measures this score? Is it RMSE as measured in linear regression?

2.) From a predictive perspective: Is such a score either good or bad? (I would guess it is not the best predicitve model on earth)

3.) What would a good score look like?

  • 1
    $\begingroup$ 1) is answered by looking at the documentation ?cv.glmnet. Scroll down to type.measure. $\endgroup$ – Sycorax May 31 '16 at 22:53
  • $\begingroup$ Thx @GeneralAbrial, so it measures deviance too, because i use the default optimization parameter, which is deviance. $\endgroup$ – Toby_Shoby May 31 '16 at 23:48
  • $\begingroup$ @GeneralAbrial, do you know how exactly cv.glmnet calculates the cvm scores when option "deviance" is chosen? Is it "deviance score"? I want to report some kind of error statistic from CV to show the expected predictive potential of my models. $\endgroup$ – Toby_Shoby Jun 1 '16 at 13:37
  • $\begingroup$ This is covered in the documentation. cvm is the out-of-sample mean of whatever metric you have chosen. $\endgroup$ – Sycorax Jun 1 '16 at 13:44

1) For logistic regression use type.measure="class" or "auc" depending on whether it is a binomial or a multinomial classification.

2) Plot the two models using a ROC curve (use ROCR package) and compare the area under the curve as shown below.

enter image description here

3) A good score would depend on the baseline you are comparing with. If your baseline is random guessing you are comparing against the purple line.

  • 1
    $\begingroup$ At best this is incomplete. Re: (1) Read type.measure in ?cv.glmnet. For logistic regression, default is deviance, but also has options class, auc, mae and mse. Perhaps you are making a recommendation to only use class or auc for logistic regression? In either case, using accuracy is an improper scoring rule, so it will select the wrong model with high probability. Re (2) models may be profitably be compared on the basis of deviance, Brier score and so on. OP is not constrained to solely compare models on the basis of AUC. $\endgroup$ – Sycorax May 31 '16 at 23:59
  • $\begingroup$ You don't need to waste space on ROC curves, as the AUC (concordance probability; $c$-index) has meaning (and better interpretation) without it. And auc may not be the best index anyway. Consider Brier score and deviance-based measures. More importantly the sample size is far too small for lasso to work reliably. Try bootstrapping the whole process and checking for volatility of features selected. $\endgroup$ – Frank Harrell Jun 12 '18 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.