I have implemented cross validation function with hyper parameter tuning. Basically, doing the following:
- Split the data into 80% training, 20% testing
- apply cross validation with hyper parameter tuning only on the 80% training data, i.e.
- apply 10-fold cross validation
- in each fold, I standardize only the training data, and apply the transformation to the validation data to avoid contamination
- after I pick the best set of hyper parameters (that ended up providing the best average accuracy in cross validation) I grab the winning hyper parameters
- re-train on the 80% training data with the set of winning hyper parameters
- test on the 20% testing data
In order to assess the performance of my data, I am interested in plotting the learning curve. For that, I grab the model with the winning set of hyper parameters and plot the learning curve for the training data ONLY (i.e.
As the learning curve internally implements cross-validation, can I pass the standardized X_train to the learning curve function, or is this prone to contamination because the training data will be split inside cv into train and validation and we might leak information from the validation data to the training data? If yes, what modifications should I do for the following:
def produce_learning_curve(self, X_train, y_train, model, model_name, nb_splits, output_folder, parameters=None, nb_repeats=None): # HERE THE X_train IS STANDARDIZED ALREADY print('Inside Learning curve') print('X_train: ', X_train.shape) print('y_train: ', y_train.shape) if nb_repeats is None: cv = StratifiedKFold(n_splits=nb_splits, random_state=42) else: cv = RepeatedStratifiedKFold(n_splits=nb_splits, n_repeats=nb_repeats, random_state=42) if parameters is None: train_sizes, train_scores, test_scores = learning_curve(model, X_train, y_train, cv=cv, scoring='accuracy') # calculate learning curve values else: train_sizes, train_scores, test_scores = learning_curve(model(**parameters), X_train, y_train, cv=cv, scoring='accuracy') # calculate learning curve values # mean of the results of the training and testing train_scores_mean = np.mean(train_scores, axis=1) test_scores_mean = np.mean(test_scores, axis=1) plt.figure() plt.xlabel("Number of Training Samples") plt.ylabel("Accuracy") plt.plot(train_sizes, train_scores_mean, label="training") plt.plot(train_sizes, test_scores_mean, label="validation") plt.legend() if not os.path.exists(output_folder): os.makedirs(output_folder) plt.savefig(output_folder + '%s_learning_curve.png' % model_name) plt.close()