How to correctly evaluate a model?

Question

I am sorry, I have a simple question that I am confused about (I AM STILL A BEGINNER):

When I create a model let's say a decision tree model and I specify random_state=integer to get reproducible outputs, then I run cross validation (let's say kfold with k=5) and I also specify random_state=integer in my CV to get reproducible outputs, then take the average R^2 for my kfolds, is this enough to give me a clue about how good is my model?

new_model = DecisionTreeRegressor(max_depth=9,
                                  min_samples_split=2,random_state=0)

crossvalidation_Decision_Trees = KFold(n_splits=5, random_state=0,shuffle=True)  
model2=new_model.fit(X_normalized, y_for_normalized)
    
scores_D_Trees = cross_val_score(model2, X_normalized,y_for_normalized, scoring='r2', cv=crossvalidation_Decision_Trees,
 n_jobs=1)
    
print("\n\nDecision Trees"+": R^2 for every fold: " + str(scores_D_Trees))

print('\033[1m'+"Decision Trees"+'\033[1m'+": Average R^2 for all the folds: " + str(np.mean(scores_D_Trees)) + '\033[0m'+ ", STD: " + str(np.std(scores_D_Trees)))

OR: Shall I remove the random_state from my decision tree model AND from my CV and let the code take different training and testing datasets every time I run the code, repeat that many times (let's say iterations=5) and at the end take the average R^2 for the average R^2 of my kfolds for these 5 iterations as an indicator for my model's performance? Will this be a better evaluation of my model?

new_model = DecisionTreeRegressor(max_depth=9,
                                      min_samples_split=2)

crossvalidation_Decision_Trees = KFold(n_splits=5,shuffle=True)  
model2=new_model.fit(X_normalized, y_for_normalized)
    
scores_D_Trees = cross_val_score(model2, X_normalized,y_for_normalized, scoring='r2', cv=crossvalidation_Decision_Trees,
 n_jobs=1)
    
print("\n\nDecision Trees"+": R^2 for every fold: " + str(scores_D_Trees))

print('\033[1m'+"Decision Trees"+'\033[1m'+": Average R^2 for all the folds: " + str(np.mean(scores_D_Trees)) + '\033[0m'+ ", STD: " + str(np.std(scores_D_Trees)))

OR: Any of these approaches is acceptable?

Note: Let's ignore hyperparameter tuning for now.

Answer 1

The scikit-learn team has written extensive tutorials on how to do cross validation well. You might want to give GridSearchCV a try. You can use it to cross-validate one model or many. This will make your code straightforward to extend when you want to use cross validation for model selection/hyperparameters tuning, as in your previous question.

Selecting dimensionality reduction with Pipeline and GridSearchCV
Cross-validation on diabetes Dataset Exercise
Comparing randomized search and grid search for hyperparameter estimation

from sklearn.datasets import load_diabetes
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeRegressor

X, y = load_diabetes(return_X_y=True)
model = DecisionTreeRegressor()

# Default parameters
# params = {}

# or

# Hard-coded parameters
params = {
    "max_depth": [9],
    "min_samples_split": [2],
}

n_splits = 10

# The KFold default is no shuffling,
# so we explicitly turn shuffling on.
cv = GridSearchCV(
    model, params, scoring="r2",
    cv=KFold(n_splits, shuffle=True),
)
cv.fit(X, y)

# Average R-squared across the k folds
cv.cv_results_["mean_test_score"]
# Standard deviation of the R-squared
cv.cv_results_["std_test_score"]