I am working on Lasso problem and the selection of the optimal tuning parameter with $k$-fold procedure, say $k=10$. Since this procedure relies on random subsampling, value of the optimal parameter will change each time I repeat the procedure. As an example, it can be 0.32, then 0.41, then 0.29, etc.

Two questions:

  1. Can I use repeated $k$-fold and average the results?
  2. How do I compute the standard error in order to use one standard rule?
  • 1
    $\begingroup$ Is it a linear model? $\endgroup$ – Dikran Marsupial Mar 19 '12 at 12:42
  • $\begingroup$ It would be good to register your account, grant. Also, I'd like to remind you that you can accept responses when you feel they directly answer your question. Registering will further allow you to get system wide notification and let you vote on Q&As, which is a sensible way to point out good replies on this site. $\endgroup$ – chl Mar 19 '12 at 13:12
  • $\begingroup$ yes it is classic linear regression with norm constraint $\endgroup$ – grant Mar 19 '12 at 22:45
  • $\begingroup$ @grant I've merged your two accounts. You can now safely use the last registered one. $\endgroup$ – chl Mar 20 '12 at 22:02
  1. Yes, repeated CV is a popular resampling technique.
  2. The sample standard deviation of your metric of interest (where one measurement corresponds to one repeat/fold combination) divided by the square root of the number of repeat-fold combinations minus one (i.e. standard error of the mean). This is done for each tuning parameter combination and then the "best" tuning parameter combination is chosen according to a certain rule (max, "one sigma rule", etc)

R package caret supports all that (including the "one sigma rule") and much more.

  • $\begingroup$ Thanks. Thus if i use 100 repeated k fold, i find 100 tunning parameters, then my optimal parameter is the average along these 100 tunning paramaeters ? $\endgroup$ – grant Mar 22 '12 at 18:27
  • $\begingroup$ No, it makes more sense to average the metrics (across repeat-fold combinations) for each tuning parameter set, and then choose the "best" tuning parameter set according to some rule applied to the average metric (max, min, "one-sigma" etc.) $\endgroup$ – Yevgeny Mar 22 '12 at 19:49
  • $\begingroup$ ok thanks. In simulations, my results are better by averaging the parameters. If i prefer use this method am i wrong ? Thanks for your answers $\endgroup$ – grant Mar 23 '12 at 9:24
  • 1
    $\begingroup$ If you have just one tuning parameter or if, in case of multiple tuning parameters, they are not correlated to each other (which is rarely the case) then your order of averaging is reasonable. If you really want to find out which tuning approach is better for your dataset/model, I would run both tuning approaches and then compare the resulting models on a holdout data set (a "validation" data set not used in training/tuning). This makes sure that you are not "over-tuning" the parameters such that the resulting model may not generalize well on other datasets. $\endgroup$ – Yevgeny Mar 23 '12 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.