0
$\begingroup$

I have read a lot about k-fold cross validation and I am getting more and more confused.

I am training neural networks and I have little training data. This suggests the use of k-fold. Up to this point, ok.

I read that k-fold will help me choose the best parameters for the networks (number of neurons, learning rate, etc.). Right?

Let's suppose that I trained networks with 50 and 100 neurons from the hidden layer. I did cross-validation and found that with 100 neurons I have better results.

And now? I have K neural networks with 100 neurons. I have K models (number of folds). How do I use this in the real world? How do I combine these models? In some places they say to carry out a new training using all the data (training and validation) with the parameters that were found to be the best. That way I would have only one model. Is that correct?

In my real work I am doing hold-out and k-fold validation. The first one says that my model gets 95% of the ratings right and with k-fold that drops to 80%. It is a very big drop.

Furthermore, it is not clear whether I can use early stop and bagging (resampling) with k-fold


Edit: Can anyone tell me if this is the correct procedure?

  1. I divide the data in training\test (80/20 for example)

  2. I use the training data with k-fold (say 10 folds). From that step, I determine which is the best algorithm and/or the best parameters for an algorithm.

  3. I train a new model with all the training data (without k-fold) using the parameters that I found to be the best in step 2.

  4. Finally, I use the test data (from step 1) to confirm that this is a good model.

Is that correct? I only use the test data at the end when I have found a single model (theoretically the best model) and I don't use it during the k-fold validation process, correct?

$\endgroup$

1 Answer 1

1
$\begingroup$

Mostly you split your dataset into train and test data. Train data is used for training and optimization, test data for a final check as a last step to verify your models quality. The latter you don't touch at all until the last step!

When optimizing hyperparameters such as the number of neurons per layer you could split train data again into train and validation data (hold-out) by random. This could have two extreme results:

  1. The model performs very well.
  2. The model performs very badly.

How do you know if it's case 1 or 2? Or something in between? You simple don't know. So how would you judge the performance (e.g. accuracy of 95%)? You can't judge it and be sure that performances will remain on this level for "real data" (test data).

Therefore you do k-fold CV. You create k splits of the train data into train and validation data and validate all of these splits. Easy case: Lets say you have k = 3 and you have two models m1 and m2 (50 neurons, 100 neurons). You split your training data 3 times, which results in 3 combinations of train and validation data. You train and validate both models m1 and m2 with all of these combinations.

M1: acc1 = 0.92, acc2 = 0.87, acc3 = 0.8
M2: acc1 = 0.98, acc2 = 0.85, acc3 = 0.91

Based on these results you can judge if m1 or m2 performs better. To do so, you calculate the mean of the accuracy for each model, so:

M1: acc_mean = 0.863
M2: acc_mean = 0.913

In this case, m2 has the better overall performance on all k splits. Now forget about all the models you have trained. All you take away is: You have found out, that 100 neurons perform better!

So you take your original train data and train a model (m2 with 100 neurons) with it. Finally you can test it with your test data.

The difference between your results with hold-out set and k-fold CV is based on choice of the data. In other words, it is not representative for performance of your model. You can try to generate 3 different hold-out sets and you will see, that some perform better then others.

Hope that helps to understand the idea of k-fold CV and interpreting the results. I suggest you to deeply understand the principles of machine learning / deep learning before moving over to early stopping and other optimization methods, otherwise you just use some techniques without knowing how they work and what their restrictions are.

$\endgroup$
3
  • $\begingroup$ Thanks for your answer. The functioning of k-fold I was able to understand. My biggest question was about which model I would use in the real world afterwards: a combination of the models resulting from the k-fold or whether I would train a new model with all the data. In practice, will this model I train later (say m2 with 100 neurons) have greater accuracy (in the test data) than the average accuracy of the models trained with k-fold (in the validation data)? $\endgroup$ Sep 15, 2020 at 18:30
  • $\begingroup$ Let's assume you found out the "best" parameters and now train a model M2 with the whole training + validation dataset. Theoretically it will achieve the average accuracy that you have seen using k-fold (e.g. acc = 0.913) also on test data. But most likely you will see different result because of two reasons: First of all, you have now trained the model with more data (train + validation instead of only train). This might lead to better results if we assume that more data can improve the results. Second, you don't know your test data. $\endgroup$
    – nopact
    Sep 18, 2020 at 9:53
  • $\begingroup$ It might happen, that your test data (as well as real world data during exploitation of the model) contains some information that have never been seen by the model while training. As a consequence you will achieve a lower accuracy. But this is what the test set is about: See if you can apply the model for real data. If the train, validation, test split are well done and the model is properly trained, you should achieve similar results. I would call this the best case. $\endgroup$
    – nopact
    Sep 18, 2020 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.