0
$\begingroup$

In glmnet using caret my the model is trained and tuned by cross validation on my training set. I have a 10 fold cross validation, over a grid of 10 alphas and 10 lambdas. The best model chosen by glmnet is the model with the combination of alpha and lambda which has the best performance. This performance is an average over the 10 cross validation folds to reduce variance.

I then use the best model to predict outcome in unseen test data and compare the predictions against the actual test set outcome.

When I call up the best model's beta coefficients (see below) am I correct in my understanding that these coefficients are the average over the 10 cross validation folds?

coef(bestModel$finalModel, bestModel$bestTune$lambda)

The training and test split is often repeated to reduce variance in calculated model performance metric. When one repeats the initial 70:30 training/test split several times in an outer loop, models with different combinations of lambda and alpha often chosen and with different beta coefficients. I know I can average the performance metric but can I also average these models beta coefficients? Given that such models commonly have different combinations of alpha and lambda.

Improve this question
$\endgroup$
6
  • $\begingroup$ How exactly do you "call up the best model's beta coefficients"? $\endgroup$ – Richard Hardy Jun 21 '17 at 7:45
  • $\begingroup$ amended question $\endgroup$ – samleighton87 Jun 21 '17 at 8:33
  • $\begingroup$ The answer then depends on what bestModel is, I suppose. But perhaps Caret documentation has something about it? $\endgroup$ – Richard Hardy Jun 21 '17 at 9:10
  • $\begingroup$ bestModel is the model chosen by cross validation from the grid of alpha and lambdas which has the best performance metric on the validation set. I'm just not sure how the coefficients are worked out - perhaps once the model is chosen it is refit on the entire test set. I'll try to clarify further. $\endgroup$ – samleighton87 Jun 21 '17 at 9:19
  • $\begingroup$ yes, you are refitting the model with the lambda and alpha found by crossvalidation. You do not average across betas, you get the new ones. $\endgroup$ – rep_ho Jun 21 '17 at 9:28
2
$\begingroup$

yes, you are refitting the model with the lambda and alpha found by the crossvalidation. You do not average across betas, you get the new ones. In the manual https://topepo.github.io/caret/model-training-and-tuning.html you can see that the algorithm refits the final model enter image description here

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you and sorry for my oversight. Are you able to answer the second part of my query - can one average the beta coefficients from different models created in an outer loop of multiple training and test sets? $\endgroup$ – samleighton87 Jun 21 '17 at 9:42
  • $\begingroup$ Sorry I missed that part. But anyway, why would you like to average beta coefficients? I can't tell you exactly why you shouldn't do it, but it sounds like an bad idea and you will probably biased your results. $\endgroup$ – rep_ho Jun 21 '17 at 9:52
  • $\begingroup$ Fair point. The reason I would like to average the coefficients is that I am trying to predict outcome in first episode psychosis and and I am interested in what variables are most important from a clinical point of view. My other idea was to chose the best performing model from the outer loop and look at it's beta coefficients for further interpretation. I am trying to get an idea what is more methodologically sound. $\endgroup$ – samleighton87 Jun 21 '17 at 9:55
  • 2
    $\begingroup$ That is a different issue altogether, more related to feature selection than coefficients (I would say). Size of a coefficient is not a good indicator of it's importance, due to collinearity, suppressor variables, moderator effects and so one. This book (free) web.stanford.edu/~hastie/StatLearnSparsity has a chapter about statistical significance in sparse models, maybe it will help you. $\endgroup$ – rep_ho Jun 21 '17 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.