I have a neural network that I use to classify data into a number of classes; in my particular case, the classes are imbalanced, but I am trying to understand this for the general case. I am using F1 score as the metric for evaluating the network. I also would like to perform K-fold cross-validation.
So, let's say I have 1000 data points and I split them into 900 training and 100 test set, which I will use for evaluation only.
I then run a K-fold cross-validation; for simplicity let's assume K=3.
I take the training set (900 elements), use the first 600 elements for training and the other 300 for validation. I then take the middle 300 elements for validation and, finally the last 300. At each fold, I will
- Save the best-performing model (according to the validation set)
- Evaluate this best-performing model on the test set.
Ok, so let's say I get these F1 scores
I have a few questions.
- Is the procedure above correct?
- When reporting the model accuracy, is it OK to just report the average, so for example, $F1 = 0.88 \pm 0.03$?
- Now, let's say I am happy with the model and I want to use it to classify some new data. Which model should I use? Should I pick the model trained in fold 3, since it has the highest score on the test data? But the one trained in fold 2, had a higher score on the validation data, so am I not biasing my choice based on the arbitrarily-picked characteristic of my test set?
- Finally, let's say that now I decide I want to change some hyperparameter or change the network structure etc. I repeat the steps above and get a "better" model, based on the evaluation on the same test set. How can I be confident that this new model is really better on a new dataset and I am not just seeing a fluke in how the test set was picked?