Machine Learning + Hyperparameter Tuning + Data Leakage : Is my procedure free of data leakage?

Question

I'm trying to classify 8 types of hand gestures with EMG signals. For that I followed these steps:

1. Split the entire data into training data and test data
2. For training data I extracted features. Here how I did that: The training dataset contains 8 files. Each file consists of 50 readings of the Myo armband for a defined gesture. Each reading includes 100 samples per sensor. There are 8 sensors. For every 100 samples/sensor, Mean (of) Absolute Values (MAV) is computed. For 8 sensors, 8 MAVS are computed for a reading. So every row contains 8 MAV values for a particular gesture. Here's a subset of training data (last column is the respective gesture number):

3. After that I tried to test several ensemble models as classifiers on the training data. For an instance, I tried to use stack of Random Forest, KNN, SVM on the training data. For that I used GridSearchCV for hyper-parameter tuning(I didn't use pipelie). Here's the code:

   param_grid = [

       {
           #Random forest
           'bootstrap': [True, False],
           'max_depth': [40, 50, 60, 70, 80],
           #'max_features': [2, 3],
           'min_samples_leaf': [3, 4, 5],
           'min_samples_split': [8, 10, 12],
           'n_estimators': [10, 15, 20, 25],
           'criterion' : ['gini', 'entropy'],
           'random_state' : [45]
       },
   
       {   
           #K Nearest Neighbours
           'n_neighbors':[5,6,7,9,11],
           'leaf_size':[1,3,5,7],
           'algorithm':['auto', 'ball_tree', 'kd_tree', 'brute'],
           'metric':['euclidean', 'manhattan'] 
   
       },
   
       {
           #SVM
           'C': list(np.arange(1, 5, 0.01)),
           'gamma': ['scale', 'auto'],
           'kernel': ['rbf', 'poly', 'sigmoid', 'linear'],
           'decision_function_shape': ['ovo', 'ovr'],
           'random_state' : [45]
       }    
   ] 
   
   models_to_train = [RandomForestClassifier(), KNeighborsClassifier(), svm.SVC()]
   
   final_models = []
   for i, model in enumerate(models_to_train):
       params = param_grid[i]
   
       clf = GridSearchCV(estimator=model, param_grid=params, cv=20, scoring = 'accuracy').fit(data_train, label_train)
       final_models.append(clf.best_estimator_)
   

4. Did similar feature extraction procedure like step 2 for test data

5. Fit the stacked model to the training data, made prediction on test data and calculated accuracy.

estimators = [
    ('rf', final_models[0]),
    ('knn', final_models[1])                 
]
clf = StackingClassifier(
    estimators=estimators, final_estimator=final_models[2]
)

category_predicted = clf.fit(data_train, label_train).predict(data_test)

acc = accuracy_score(label_test, category_predicted) * 100

Now, my question is,

Is there any chance of data leakage in this procedure?

Edit

I believe this procedure suffers from data leakage because I did feature extraction in step 2 on entire training data and those features are used in GridSearchCV without any pipeline . If I put feature extraction (what described in step 2) and estimator in pipeline (as discussed here: https://towardsdatascience.com/pre-process-data-with-pipeline-to-prevent-data-leakage-during-cross-validation-e3442cca7fdc), then it can be avoided.

Answer 1

A potentially bigger problem than data leakage between test and training sets is the unreliability of test/train splits of small data sets. You need many thousands of cases for that to be reliable. Otherwise you are throwing away information by limiting the size of the training set, and you are getting imprecise estimates of model validity by having too small a test set.

As you have a well-defined algorithm for building your model, consider internal validation by bootstrapping instead. This answer outlines the procedure. Even if you continue to use a train/test split for your modeling, repeating the entire modeling process including the original train/test split on multiple bootstrap samples of your data will evaluate how much of a problem is imposed by any data leakage.

In your case you don't appear to have a problem with data leak in your cross validation. All you have done is to combine raw readings into a type of average, the MAV, without any attempt to standardize the readings within each sensor at that point of the analysis. There might be some standardization later on within your parameter search, but so far as I can tell (I'm not fluent in sklearn) that seems to be done appropriately.

That's different from the situation described in the page you link. There each of the predictors was standardized from the beginning to put them on the same relative scale, as is needed for principal-component analyses and penalized regressions (ridge, LASSO). As the degree of transformation of any predictor needed for the standardization will vary from sample to sample, that can be a problem if your (wise) intent is to repeat the entire modeling process (including the standardization) in each CV fold or bootstrap sample. You haven't fed your parameter search pre-standardized data, just an average without any change of scale, so you shouldn't have that problem.