In K fold cross validation, we divide the dataset into k folds, where we train the model on k-1 folds and test the model on the remaining fold. We do so until all the folds were assigned as the test set. In every of these iterations, we will train a new model that is independent of the model created from the previous iteration ( every iteration uses a new instance of the model). So my question is, if I divide my dataset into train and test sets, then I have used only the training set for the k cross validation process, and since every iteration uses a new model, what is the output model from this k fold cross validation process that I should use to evaluate it ( calculates the ROC curve, F1-score, precision and so on) using the test set ?? (As I have different models for every iteration). One way to implement k fold cross validation is to use sklearn.model_selection.cross_val_score and this returns only an array of scores of the model for each run of the cross validation and this confirms my problem, where there is no model is returned to be further evaluated by the test set. What should I do in this case ?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ It is probably more common to call these sets "training" and "validation", not "training" and "test". A test set is one that is set completely aside when the cross validation is being carried out on the {training+validation} set. $\endgroup$– Richard HardyCommented Nov 20, 2022 at 17:54
-
$\begingroup$ @RichardHardy Yes and this is actually what I mean. I mean " Test set", where I want to use it after the cross validation process to predict the model's performance using precision, recall, F1-score and AUC. $\endgroup$– AAACommented Nov 20, 2022 at 18:17
-
1$\begingroup$ Relevant answer here on how data-splitting should be practiced for CV and hyper-parameter optimisation: from the question Does using grid search for hyperparemeters make test set redundant? $\endgroup$– patagonicusCommented Nov 20, 2022 at 19:41
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
If you use K-fold cross validation (CV) for hyper parameter tuning, you should train a single model on the entire training set with the best found hyper-parameters and test on the test set.
If you use K-fold CV for performance evaluation (like in sklearn cross_val_score), then you don't need to split your dataset into train/test. The performance reported in each fold will be a test performance. People usually average them or get all the predictions and then evaluate the entire dataset. This is usually done to assess performance when the dataset is small and there isn't a single model output for this case, nor the aim is to have it.
-
$\begingroup$ Thanks alot. So to make sure that I have understand you correctly. If my target is to optimize hyperparameters, then I train using k fold cross validation and the train dataset while I am finding the best parameters, then when I find them, I train the model again using the best parameters and then evaluate using the test set. On the other hand, If I need to evaluate the model only, then I use the whole dataset as k-fold cross validation in this case will do the splitting internally and the final performance is the average performance of all the iterations. Is that correct ? $\endgroup$– AAACommented Nov 20, 2022 at 16:55
-
$\begingroup$ That is correct. In the second one, an alternative method (as described above) is to get all the predictions for the entire dataset and calculate the performance (especially if fold sizes are small). $\endgroup$– gunesCommented Nov 20, 2022 at 17:02