I've recently gotten into ML and I'm a bit confused about rep k fold cross validation, train, test split and overfitting. I have already read some of the posts in this forum, but none of them could answer my question exactly.
A simple train test split is used in most of the posts on the internet. But k fold cross validation or even rep k fold cross validation is often used to validate the model. That doesn't make any sense to me. So wouldn't it make more sense to train and test the model with rep k fold cross validation every time and to calculate the mean values of the probabilities for e.g. a logistic regression? Or does that lead to overfitting? Even if the selection is random, a one-time train test split could lead to inaccurate results, right? In the following I have described the two scenarios with codes. I found out that rep k fold cross validation doesn't support cross_val_predict attribute. Note: I haven't calculated the mean values of the probabilities in list_with_proba yet.
Thanks in advance!
from sklearn.linear_model import LogisticRegression X_train, X_test, y_train, y_test = train_test_split(x, y, train_size = 0.8, random_state = 42) model = LogisticRegression() model.fit(X_train, y_train) model.predict_proba(X_train) # VS. from sklearn.model_selection import KFold from sklearn.model_selection import cross_val_score from sklearn.model_selection import cross_val_predict from sklearn.linear_model import LogisticRegression list_with_scores =  list_with_proba =  repeats = 10 splits = 5 for i in range(repeats): model = LogisticRegression() kf = KFold(n_splits = splits, shuffle = True) scores = cross_val_score(model, X, y, scoring = "accuracy", cv = kf) list_with_scores.append(scores) y_predict_proba = cross_val_predict(model, X, y, method = "predict_proba", cv = kf) list_with_proba.append(y_predict_proba) ```