Is it logical to combine cross-validation estimator like RidgeCV with cross_val_score in sklearn?

Question

I was going through solutions for a regression problem competition on Kaggle here. Many solutions for the problem are combining cross-validation estimators like RidgeCV, LassoCV with cross_val_score for the model selection phase (Later, they blend these models). One example from a solution is

kfolds = KFold(n_splits=10, shuffle=True, random_state=42)

# model scoring and validation function
def cv_rmse(model, X=X):
    rmse = np.sqrt(-cross_val_score(model, X, y,scoring="neg_mean_squared_error",cv=kfolds))
    return (rmse)

# rmsle scoring function
def rmsle(y, y_pred):
    return np.sqrt(mean_squared_error(y, y_pred))

# This is a range of values that the model considers each time in runs a CV
e_alphas = [0.0001, 0.0002, 0.0003, 0.0004, 0.0005, 0.0006, 0.0007]
e_l1ratio = [0.8, 0.85, 0.9, 0.95, 0.99, 1]
alphas_alt = [14.5, 14.6, 14.7, 14.8, 14.9, 15, 15.1, 15.2, 15.3, 15.4, 15.5]
alphas2 = [5e-05, 0.0001, 0.0002, 0.0003, 0.0004, 0.0005, 0.0006, 0.0007, 0.0008]


# Kernel Ridge Regression : made robust to outliers
ridge = make_pipeline(RobustScaler(), RidgeCV(alphas=alphas_alt, cv=kfolds))

# LASSO Regression : made robust to outliers
lasso = make_pipeline(RobustScaler(), LassoCV(max_iter=1e7, 
                    alphas=alphas2,random_state=42, cv=kfolds))

# Elastic Net Regression : made robust to outliers
elasticnet = make_pipeline(RobustScaler(), ElasticNetCV(max_iter=1e7, 
                         alphas=e_alphas, cv=kfolds, l1_ratio=e_l1ratio))

# store models, scores and prediction values 
models = {'Ridge': ridge,
          'Lasso': lasso, 
          'ElasticNet': elasticnet}
predictions = {}
scores = {}

for name, model in models.items():
    
    model.fit(X, y)
    predictions[name] = np.expm1(model.predict(X))
    
    score = cv_rmse(model, X=X)
    scores[name] = (score.mean(), score.std())

There are multiple solution posts like this. But does this use make any sense? In other words, my primary question is - Isn't this wasteful/illogical to fit the model first in the loop directly and then doing the same thing inside the cv_rmse function, and that too with a separate cross_val_score? Maybe I am confused on what {cross-validation-estimator}.fit like RidgeCV.fit actually does. I understand that it helps in finding best parameters like GridSearchCV, but does it then automatically fit the whole train data with the best parameters to be then used directly for predictions and scoring? Or do we need to access the best parameters and then retrain using those parameters? The documentation is also not very clear on this.

A secondary question is - does it ever make sense to predict on the same X which it was trained on as done in the for loop in the end? Or are the predictions here like done in cross_val_predict.(which also wouldn't make sense)

I'm new here. If I have to post the secondary question separately, please suggest so and focus on the primary question.

Answer 1

You're right that this is poorly documented. As this Github issue mentions and this line of code suggests, it uses the refit mechanism of GridSearchCV (see here, refit is True by default), i.e. when it's found the best hyper-parameter (HP), it fits the model to the entire training data.

Using cross_val_predict together with CV models is used for convenience when you want to do nested cross-validation. There is no single model at the end, but there are predictions and the associated performance. Outer loop is responsible of making predictions for the holdout set, where the inner loop is responsible for choosing the best model for the given training set. So, it makes sense in the end.

There is little data-leakage in the implementation with how the data is preprocessed, RobustScaler calculates median and mean absolute deviation using train+validation sets. But, this occurs when choosing the best HPs, not making predictions.