Timeline for after using cross-validation, is a separate train-test split necessary for generating a model?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 17, 2017 at 11:00 | comment | added | cbeleites | Also, in using CV for hyperparameter optimization, the CV itself does not return the model. Within the optimization, CV typoiclly just returns some figure of merit for every model (parameter/hyperparameter set) that is fed into the CV. It is a separate point how to decide with the help of these figures of merit which (hyper)parameters to use for the optimized model. And even if the default is "take the one that looked best" that is neither the only possible nor the only sensibe option. | |
| May 17, 2017 at 10:54 | comment | added | cbeleites | ... be obtained in a number of ways, including techiques that validate/verify performance of a given model. However, using validation techiques internally during part of the training process (optimization is still part of training!) does not get rid of the need of doing a proper validation (verification) with the final model. And there again you have the choice among a number of different techniques, including cross validation. | |
| May 17, 2017 at 10:50 | comment | added | cbeleites | There's IMHO a lot of confusion coming from the particular use of vocabulary here (= in this field). Cross validation in itself is independent of whether its results are used for determining hyperparameters or for, well, validation (or more precisely, verification) of generalization error. So in that context, the book sentence is exactly right. But a lot of confusion comes from the use of a technique with "validation" in its name for other (search/optimization) purposes. The point here is, model optimization relies on optimizing some figure of merit = some measure of performance. This may... | |
| May 17, 2017 at 0:04 | comment | added | Stephen | Oh ok thanks, I didn't realize you were referring to the impact of hyperparameters on regular parameters. I just didn't get your terminology but it makes sense now. | |
| May 16, 2017 at 6:14 | comment | added | Cagdas Ozgenc | It doesn't reduce in the sense of removing them, but hyperparameters limit the free movement of the parameters of the model during the optimization. For example in ridge penalty, the total squared weights are bounded by lambda hyperparameter. | |
| May 16, 2017 at 1:57 | comment | added | Stephen | Sorry, let me rephrase my question to be clear: I don't understand what you mean by "reduce the number of freely adjusting parameters." I don't see why cross-validation "reduces" parameters, I thought it just "chooses" them. | |
| May 14, 2017 at 19:37 | comment | added | Cagdas Ozgenc | Everytime you adjust and test a parameter on the same data set you are introducing a bias. Model parameters are set and evaluated on training set hence create a bias. Hyperparameters are set and evaluated on the validation set hence create a bias but lower. I don't know what you mean by not using parameters. Without parameters there's nothing to train. | |
| May 14, 2017 at 19:12 | comment | added | Stephen | I do have one question about your answer: You say that cross-validation (or perhaps "grid search with cross validation") is done to reduce the number of freely adjusting parameters. My understanding so far is that this is done to choose such parameters. This process will produce bias because of the likely overfitting of these hyperparameters. That part I understand, but is there something more you're referring to here other than the evaluation of many parameters to choose the best one? Perhaps the decision not to use some parameters at all? | |
| May 14, 2017 at 18:52 | comment | added | Stephen | Upvoted for the thoughtful answer. I am still wrapping my head around these concepts and going through the book. My goal is to accept an answer once I understand everything better, since the two answers are somewhat different in emphasis. | |
| May 12, 2017 at 8:03 | history | edited | Cagdas Ozgenc | CC BY-SA 3.0 |
added 7 characters in body
|
| May 12, 2017 at 7:50 | history | edited | Cagdas Ozgenc | CC BY-SA 3.0 |
added 18 characters in body
|
| May 12, 2017 at 7:32 | history | answered | Cagdas Ozgenc | CC BY-SA 3.0 |