I want to build classification machine learning using logistic regression. First i check my model roc_auc score using train test split and check with k cross validation. Turns out there is indication that my model is overfit.. so i decided to do hyperparameter tuning.
After i got some best parameter from hyperparameter tuning using grid search cv. Should i check my model roc_auc score once more with k cross validation?
You already used cross-validation during the grid search, so it won't make much sense to run cross-validation again. What would make sense is to split the data into train and test samples before tuning the parameters, then use cross-validation on the train set, and after finding the hyperparameters, test the model on the held-out test set.
Cross-validation provides information about how well a classifier generalizes, specifically the range of expected errors of the classifier.
When looking for the best hyperparameters a way to generalise that the learning has been correct under the different values tested, then there is no need to cross validate again as this error is representative.
So once you have found the best hyperparameters and you know the most representative error metric computed with CV, if you want to put the model into production, then you must now train with the full available data set to take advantage of this information.
One remark about the data set size that could be of interest for you:
However, a classifier trained on a high dimensional dataset with no structure may still perform better than expected on cross-validation, just by chance. This can typically happen with small datasets with less than a few hundred samples. permutation_test_score provides information on whether the classifier has found a real class structure and can help in evaluating the performance of the classifier.
It is important to note that this test has been shown to produce low p-values even if there is only weak structure in the data because in the corresponding permutated datasets there is absolutely no structure. This test is therefore only able to show when the model reliably outperforms random guessing.
Finally, permutation_test_score is computed using brute force and internally fits (n_permutations + 1) * n_cv models. It is therefore only tractable with small datasets for which fitting an individual model is very fast.
Source:Cross validation sklearn
