rep k fold cross validation, train test split and overfitting

Question

I've recently gotten into ML and I'm a bit confused about rep k fold cross validation, train, test split and overfitting. I have already read some of the posts in this forum, but none of them could answer my question exactly.

A simple train test split is used in most of the posts on the internet. But k fold cross validation or even rep k fold cross validation is often used to validate the model. That doesn't make any sense to me. So wouldn't it make more sense to train and test the model with rep k fold cross validation every time and to calculate the mean values ​​of the probabilities for e.g. a logistic regression? Or does that lead to overfitting? Even if the selection is random, a one-time train test split could lead to inaccurate results, right? In the following I have described the two scenarios with codes. I found out that rep k fold cross validation doesn't support cross_val_predict attribute. Note: I haven't calculated the mean values ​​of the probabilities in list_with_proba yet.

Thanks in advance!

from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(x, y, train_size = 0.8, random_state = 42)

model = LogisticRegression()

model.fit(X_train, y_train)

model.predict_proba(X_train)


# VS.


from sklearn.model_selection import KFold

from sklearn.model_selection import cross_val_score

from sklearn.model_selection import cross_val_predict

from sklearn.linear_model import LogisticRegression


list_with_scores = []

list_with_proba = []

repeats = 10

splits = 5

for i in range(repeats):
    
    model = LogisticRegression()
    
    
    kf = KFold(n_splits = splits, shuffle = True)
    
    scores = cross_val_score(model, X, y, scoring = "accuracy", cv = kf)
    
    list_with_scores.append(scores)
    
    
    y_predict_proba = cross_val_predict(model, X, y, method = "predict_proba", cv = kf)
    
    list_with_proba.append(y_predict_proba) ```

Answer 1

The key point to consider here is that the reason for the primary train/test split is the test dataset should not be used at any point in the model selection/training process. It's kept to one side, and only used once the final model has been built. It is then used to test that model to evaluate how well the model should perform when deployed. This strategy helps ensure the model does not overfit the test data, so the evaluation results should be representative of how the model performs when deployed (assuming of course that the test data is representative of "real" data).

Attempting to use k-fold cross validation and then averaging the results (ensembling) as you suggest means that all the data is used during model training, so there is no independent test set that can be used for model evaluation. While using the strategy you suggest could lead to a better model - both having more training data and ensembling multiple results should result in a better performing model - the lack of an independent test set means you can't tell if the model really is better, or if it just overfitted to the test data.

K-fold cross validation is used on the training set, usually either for hyperparameter tuning or for model selection. However, I don't see any reason why you can't build an ensemble using k-fold splits of the training set to generate the training data for each ensemble member. You then still have the test data to evaluate your ensemble model.