0
$\begingroup$

I want to use repeated Kfold cross validation in my experiment, since I have a small dataset that might be prone to fluctuation in results of a regular cross valudation regime, so I am opting for a repeated one.

Basically I want to make sure if the following process is correct (Lets assume I want to have a 10-fold cross validation, repeated 10 times...):

  • split the data into 80% training and 20% testing
  • apply repeated k-fold cross validation only on the 80% training.
  • first loop is the hyper parameters loop (over the set of all possible hyper params)
  • inside this loop, I have the repeated cross validation loop (I am using RepeatedKFold from sklearn)
  • at the end, do I end up with 10 * N sets of hyper parameters (Let N denote the number of possible hyper parameters I am looping on, and 10 is the number of repeats)?
  • Do I choose the best out of those 10*N hyper parameters???? and then:
  • retrain on the whole training data again using the selected set of hyper parameters
  • test on the remaining 20% of data

I am using the following code so far:

def cross_validation(df, nb_splits, split_ratio: float, nb_repeats=None):
    X = np.array(df.loc[:, df.columns != 'demand'])
    y = np.array(df.loc[:, 'demand'])
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_ratio, random_state=42)

    tempModels = []
    if nb_repeats is None:
        kf = KFold(n_splits=nb_splits)
    else:
        kf = RepeatedKFold(n_splits=nb_splits, n_repeats=nb_repeats, random_state=2652124)
    parameters = {'alpha': [1, 0.1, 0.01, 0.001, 0.0001, 0]}

    def get_param_grid(dicts):
        return [dict(zip(dicts.keys(), p)) for p in it.product(*dicts.values())]

    for parameter in get_param_grid(parameters):

        model = Lasso(**parameter)
        r2_scores, adj_r2_scores, rmse_scores, mse_scores, mae_scores, mape_scores = [], [], [], [], [], []

        for train_index, test_index in kf.split(X_train):
            X_train_inner, X_val = X_train[train_index], X_train[test_index]
            y_train_inner, y_val = y_train[train_index], y_train[test_index]

            ''' standardization inside k-fold cross validation, to avoid contamination '''
            scaler = StandardScaler()
            X_train_inner = scaler.fit_transform(X_train_inner)
            X_val = scaler.transform(X_val)

            model.fit(X_train_inner, y_train_inner)
            y_pred = model.predict(X_val)
            r2, adj_r2, rmse, mse, mae, mape = get_stats(y_val, y_pred, X_val.shape[1])

            r2_scores.append(r2)
            adj_r2_scores.append(adj_r2)
            rmse_scores.append(rmse)
            mse_scores.append(mse)
            mae_scores.append(mae)
            mape_scores.append(mape)

        tempModels.append(
            [parameter, np.mean(r2_scores), np.mean(adj_r2_scores), np.mean(rmse_scores), np.mean(mse_scores),
             np.mean(mae_scores), np.mean(mape_scores)])

    tempModels = sorted(tempModels, key=lambda k: k[3])
    best_hyperparams = tempModels[0][0]
    print('Best Validation Scores:\nR^2: %.3f\nAdj R^2: %.3f\nRMSE: %.3f\nMSE: %.3f\nMAE: %.3f\nMAPE: %.3f\n' %
          (tempModels[0][1], tempModels[0][2], tempModels[0][3], tempModels[0][4], tempModels[0][5], tempModels[0][6]))

    model = Lasso(**tempModels[0][0])

    scaler = StandardScaler()
    X_train = scaler.fit_transform(X_train)
    X_test = scaler.transform(X_test)

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    r2, adj_r2, rmse, mse, mae, mape = get_stats(y_test, y_pred, X_test.shape[1])
    print('Testing Scores:\nR^2: %.3f\nAdj R^2: %.3f\nRMSE: %.3f\nMSE: %.3f\nMAE: %.3f\nMAPE: %.3f\n' %
          (r2, adj_r2, rmse, mse, mae, mape))
    print('average demand')

Thank you in advance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.