I want to use repeated Kfold cross validation in my experiment, since I have a small dataset that might be prone to fluctuation in results of a regular cross valudation regime, so I am opting for a repeated one.

Basically I want to make sure if the following process is correct (Lets assume I want to have a 10-fold cross validation, repeated 10 times...):

  • split the data into 80% training and 20% testing
  • apply repeated k-fold cross validation only on the 80% training.
  • first loop is the hyper parameters loop (over the set of all possible hyper params)
  • inside this loop, I have the repeated cross validation loop (I am using RepeatedKFold from sklearn)
  • at the end, do I end up with 10 * N sets of hyper parameters (Let N denote the number of possible hyper parameters I am looping on, and 10 is the number of repeats)?
  • Do I choose the best out of those 10*N hyper parameters???? and then:
  • retrain on the whole training data again using the selected set of hyper parameters
  • test on the remaining 20% of data

I am using the following code so far:

def cross_validation(df, nb_splits, split_ratio: float, nb_repeats=None):
    X = np.array(df.loc[:, df.columns != 'demand'])
    y = np.array(df.loc[:, 'demand'])
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_ratio, random_state=42)

    tempModels = []
    if nb_repeats is None:
        kf = KFold(n_splits=nb_splits)
        kf = RepeatedKFold(n_splits=nb_splits, n_repeats=nb_repeats, random_state=2652124)
    parameters = {'alpha': [1, 0.1, 0.01, 0.001, 0.0001, 0]}

    def get_param_grid(dicts):
        return [dict(zip(dicts.keys(), p)) for p in it.product(*dicts.values())]

    for parameter in get_param_grid(parameters):

        model = Lasso(**parameter)
        r2_scores, adj_r2_scores, rmse_scores, mse_scores, mae_scores, mape_scores = [], [], [], [], [], []

        for train_index, test_index in kf.split(X_train):
            X_train_inner, X_val = X_train[train_index], X_train[test_index]
            y_train_inner, y_val = y_train[train_index], y_train[test_index]

            ''' standardization inside k-fold cross validation, to avoid contamination '''
            scaler = StandardScaler()
            X_train_inner = scaler.fit_transform(X_train_inner)
            X_val = scaler.transform(X_val)

            model.fit(X_train_inner, y_train_inner)
            y_pred = model.predict(X_val)
            r2, adj_r2, rmse, mse, mae, mape = get_stats(y_val, y_pred, X_val.shape[1])


            [parameter, np.mean(r2_scores), np.mean(adj_r2_scores), np.mean(rmse_scores), np.mean(mse_scores),
             np.mean(mae_scores), np.mean(mape_scores)])

    tempModels = sorted(tempModels, key=lambda k: k[3])
    best_hyperparams = tempModels[0][0]
    print('Best Validation Scores:\nR^2: %.3f\nAdj R^2: %.3f\nRMSE: %.3f\nMSE: %.3f\nMAE: %.3f\nMAPE: %.3f\n' %
          (tempModels[0][1], tempModels[0][2], tempModels[0][3], tempModels[0][4], tempModels[0][5], tempModels[0][6]))

    model = Lasso(**tempModels[0][0])

    scaler = StandardScaler()
    X_train = scaler.fit_transform(X_train)
    X_test = scaler.transform(X_test)

    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    r2, adj_r2, rmse, mse, mae, mape = get_stats(y_test, y_pred, X_test.shape[1])
    print('Testing Scores:\nR^2: %.3f\nAdj R^2: %.3f\nRMSE: %.3f\nMSE: %.3f\nMAE: %.3f\nMAPE: %.3f\n' %
          (r2, adj_r2, rmse, mse, mae, mape))
    print('average demand')

Thank you in advance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.