Standardize data before plotting learning curve?

Question

I have implemented cross validation function with hyper parameter tuning. Basically, doing the following:

• Split the data into 80% training, 20% testing
• apply cross validation with hyper parameter tuning only on the 80% training data, i.e.
  • apply 10-fold cross validation
  • in each fold, I standardize only the training data, and apply the transformation to the validation data to avoid contamination
• after I pick the best set of hyper parameters (that ended up providing the best average accuracy in cross validation) I grab the winning hyper parameters
• re-train on the 80% training data with the set of winning hyper parameters
• test on the 20% testing data

Question

In order to assess the performance of my data, I am interested in plotting the learning curve. For that, I grab the model with the winning set of hyper parameters and plot the learning curve for the training data ONLY (i.e. X_train and y_train)

As the learning curve internally implements cross-validation, can I pass the standardized X_train to the learning curve function, or is this prone to contamination because the training data will be split inside cv into train and validation and we might leak information from the validation data to the training data? If yes, what modifications should I do for the following:

    def produce_learning_curve(self, X_train, y_train, model, model_name, nb_splits, output_folder, parameters=None, nb_repeats=None):

        # HERE THE X_train IS STANDARDIZED ALREADY 

        print('Inside Learning curve')
        print('X_train: ', X_train.shape)
        print('y_train: ', y_train.shape)

        if nb_repeats is None:
            cv = StratifiedKFold(n_splits=nb_splits, random_state=42)
        else:
            cv = RepeatedStratifiedKFold(n_splits=nb_splits, n_repeats=nb_repeats, random_state=42)

        if parameters is None:
            train_sizes, train_scores, test_scores = learning_curve(model, X_train, y_train, cv=cv, scoring='accuracy')  # calculate learning curve values
        else:
            train_sizes, train_scores, test_scores = learning_curve(model(**parameters), X_train, y_train, cv=cv, scoring='accuracy')  # calculate learning curve values

        # mean of the results of the training and testing
        train_scores_mean = np.mean(train_scores, axis=1)
        test_scores_mean = np.mean(test_scores, axis=1)

        plt.figure()
        plt.xlabel("Number of Training Samples")
        plt.ylabel("Accuracy")

        plt.plot(train_sizes, train_scores_mean, label="training")
        plt.plot(train_sizes, test_scores_mean, label="validation")
        plt.legend()

        if not os.path.exists(output_folder):
            os.makedirs(output_folder)
        plt.savefig(output_folder + '%s_learning_curve.png' % model_name)
        plt.close()

Answer 1

After a bit of research I found out that I can use pipeline in order to avoid data leakage.

references: this and this

This is how my code looks like now:

    def produce_learning_curve(self, model, model_name, nb_splits, output_folder, parameters=None, nb_repeats=None):

        X_train, y_train = self.X_train, self.y_train

        pipe = Pipeline([
            ('sc', StandardScaler()),
            ('model', model(**parameters, random_state=42))
        ])

        print('Inside Learning curve')
        print('X_train: ', self.X_train.shape)
        print('y_train: ', self.y_train.shape)

        if nb_repeats is None:
            cv = StratifiedKFold(n_splits=nb_splits, random_state=42)
        else:
            cv = RepeatedStratifiedKFold(n_splits=nb_splits, n_repeats=nb_repeats, random_state=42)

        # train_sizes, train_scores, test_scores = learning_curve(model(**parameters), X_train, y_train, cv=cv, scoring='accuracy')  # calculate learning curve values

        train_sizes, train_scores, test_scores = learning_curve(pipe, X_train, y_train, cv=cv, scoring='accuracy')  # calculate learning curve values

        # mean of the results of the training and testing
        train_scores_mean = np.mean(train_scores, axis=1)
        test_scores_mean = np.mean(test_scores, axis=1)

        plt.figure()
        plt.xlabel("Number of Training Samples")
        plt.ylabel("Accuracy")

        plt.plot(train_sizes, train_scores_mean, label="training")
        plt.plot(train_sizes, test_scores_mean, label="validation")
        plt.legend()

        if not os.path.exists(output_folder):
            os.makedirs(output_folder)
        plt.savefig(output_folder + '%s_learning_curve.png' % model_name)
        plt.close()
```