I've split my data into a training and test set (75% and 25% split respectively), and have subsequently performed a 5-fold cross validation on the training set using GridSearchCV in order to find the optimal hyperparameters for four different classification models (LR, RF, SVM and k-NN). I have then yielded their corresponding validation, training and testing AUC scores.
However, I am confused about which AUC score to pay attention to when it comes to describing the performance of the classifiers as well as deciding which one is the most optimal one - do I pay attention to the validation AUC score or to the test AUC score? Should it be the former case, what is the point of splitting the data into a test and training set in the first place?
There are countless examples online where data splitting is always performed, where the mean validation AUC and test AUC scores are found, and where the ROC curve is drawn for the test set, but everybody always seems to neglect the explanation of which AUC score one should focus on when choosing the optimal classification model, and most importantly, why. Does the test set act as some form of reassurance?
In my situation I obtain the following AUC scores:
SVM training AUC: 0.727 SVM validation AUC: 0.703 SVM test AUC: 0.762 RF training AUC: 1.000 RF validation AUC: 0.791 RF test AUC: 0.625 LR training AUC: 0.776 LR validation AUC: 0.689 LR test AUC: 0.737 k-NN training AUC: 0.895 k-NN validation AUC: 0.792 k-NN test AUC: 0.646