1
$\begingroup$

I have a neural network that I use to classify data into a number of classes; in my particular case, the classes are imbalanced, but I am trying to understand this for the general case. I am using F1 score as the metric for evaluating the network. I also would like to perform K-fold cross-validation.

So, let's say I have 1000 data points and I split them into 900 training and 100 test set, which I will use for evaluation only.

I then run a K-fold cross-validation; for simplicity let's assume K=3.

I take the training set (900 elements), use the first 600 elements for training and the other 300 for validation. I then take the middle 300 elements for validation and, finally the last 300. At each fold, I will

  1. Save the best-performing model (according to the validation set)
  2. Evaluate this best-performing model on the test set.

Ok, so let's say I get these F1 scores

fold train F1 val F1 test F1
1 0.93 0.88 0.87
2 0.95 0.93 0.86
3 0.94 0.91 0.92
Average 0.94 0.91 0.88
SD 0.01 0.03 0.03

I have a few questions.

  1. Is the procedure above correct?
  2. When reporting the model accuracy, is it OK to just report the average, so for example, $F1 = 0.88 \pm 0.03$?
  3. Now, let's say I am happy with the model and I want to use it to classify some new data. Which model should I use? Should I pick the model trained in fold 3, since it has the highest score on the test data? But the one trained in fold 2, had a higher score on the validation data, so am I not biasing my choice based on the arbitrarily-picked characteristic of my test set?
  4. Finally, let's say that now I decide I want to change some hyperparameter or change the network structure etc. I repeat the steps above and get a "better" model, based on the evaluation on the same test set. How can I be confident that this new model is really better on a new dataset and I am not just seeing a fluke in how the test set was picked?
$\endgroup$

1 Answer 1

2
$\begingroup$

1 and 2 look good!

For 3, you should choose the model based off of validation set results. Since you’ve done multiple folds, you can take the hyper parameters that returned the highest average validation accuracy as best. The test set should be a complete hold out as an estimate to see how it will do when deployed. Another common practice is once you have run cross-validation, you would retrain the neural network (with the best hyperparameters) on the training+validation data, since the more data fed into these models, the better.

For 4, You would do the same process as above, and only use the test set as a gauge, rather than using it to choose a better model.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much, this makes a lot of sense! $\endgroup$
    – nico
    Jul 6, 2023 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.