I've split my data into a training and test set (75% and 25% split respectively), and have subsequently performed a 5-fold cross validation on the training set using GridSearchCV in order to find the optimal hyperparameters for four different classification models (LR, RF, SVM and k-NN). I have then yielded their corresponding validation, training and testing AUC scores.
However, I am confused about which AUC score to pay attention to when it comes to describing the performance of the classifiers as well as deciding which one is the most optimal one - do I pay attention to the validation AUC score or to the test AUC score? Should it be the former case, what is the point of splitting the data into a test and training set in the first place?
There are countless examples online where data splitting is always performed, where the mean validation AUC and test AUC scores are found, and where the ROC curve is drawn for the test set, but everybody always seems to neglect the explanation of which AUC score one should focus on when choosing the optimal classification model, and most importantly, why. Does the test set act as some form of reassurance?
In my situation I obtain the following AUC scores:
SVM training AUC: 0.727
SVM validation AUC: 0.703
SVM test AUC: 0.762
RF training AUC: 1.000
RF validation AUC: 0.791
RF test AUC: 0.625
LR training AUC: 0.776
LR validation AUC: 0.689
LR test AUC: 0.737
k-NN training AUC: 0.895
k-NN validation AUC: 0.792
k-NN test AUC: 0.646