# Machine Learning + Hyperparameter Tuning + Data Leakage : Is my procedure free of data leakage?

I'm trying to classify 8 types of hand gestures with EMG signals. For that I followed these steps:

1. Split the entire data into training data and test data
2. For training data I extracted features. Here how I did that: The training dataset contains 8 files. Each file consists of 50 readings of the Myo armband for a defined gesture. Each reading includes 100 samples per sensor. There are 8 sensors. For every 100 samples/sensor, Mean (of) Absolute Values (MAV) is computed. For 8 sensors, 8 MAVS are computed for a reading. So every row contains 8 MAV values for a particular gesture. Here's a subset of training data (last column is the respective gesture number):

1. After that I tried to test several ensemble models as classifiers on the training data. For an instance, I tried to use stack of Random Forest, KNN, SVM on the training data. For that I used GridSearchCV for hyper-parameter tuning(I didn't use pipelie). Here's the code:

param_grid = [

    {
#Random forest
'bootstrap': [True, False],
'max_depth': [40, 50, 60, 70, 80],
#'max_features': [2, 3],
'min_samples_leaf': [3, 4, 5],
'min_samples_split': [8, 10, 12],
'n_estimators': [10, 15, 20, 25],
'criterion' : ['gini', 'entropy'],
'random_state' : [45]
},

{
#K Nearest Neighbours
'n_neighbors':[5,6,7,9,11],
'leaf_size':[1,3,5,7],
'algorithm':['auto', 'ball_tree', 'kd_tree', 'brute'],
'metric':['euclidean', 'manhattan']

},

{
#SVM
'C': list(np.arange(1, 5, 0.01)),
'gamma': ['scale', 'auto'],
'kernel': ['rbf', 'poly', 'sigmoid', 'linear'],
'decision_function_shape': ['ovo', 'ovr'],
'random_state' : [45]
}
]

models_to_train = [RandomForestClassifier(), KNeighborsClassifier(), svm.SVC()]

final_models = []
for i, model in enumerate(models_to_train):
params = param_grid[i]

clf = GridSearchCV(estimator=model, param_grid=params, cv=20, scoring = 'accuracy').fit(data_train, label_train)
final_models.append(clf.best_estimator_)

2. Did similar feature extraction procedure like step 2 for test data

3. Fit the stacked model to the training data, made prediction on test data and calculated accuracy.

estimators = [
('rf', final_models[0]),
('knn', final_models[1])
]
clf = StackingClassifier(
estimators=estimators, final_estimator=final_models[2]
)

category_predicted = clf.fit(data_train, label_train).predict(data_test)

acc = accuracy_score(label_test, category_predicted) * 100


Now, my question is,

Is there any chance of data leakage in this procedure?

Edit

I believe this procedure suffers from data leakage because I did feature extraction in step 2 on entire training data and those features are used in GridSearchCV without any pipeline . If I put feature extraction (what described in step 2) and estimator in pipeline (as discussed here: https://towardsdatascience.com/pre-process-data-with-pipeline-to-prevent-data-leakage-during-cross-validation-e3442cca7fdc), then it can be avoided.

• I'm not completely sure what you mean by "MAV" in your Step 2, where you say that you did feature extraction. Please provide more details on that.
– EdM
Aug 24 '20 at 22:32
• I added definition of MAV (Mean (of) Absolute Values) in question. Aug 24 '20 at 22:53

In your case you don't appear to have a problem with data leak in your cross validation. All you have done is to combine raw readings into a type of average, the MAV, without any attempt to standardize the readings within each sensor at that point of the analysis. There might be some standardization later on within your parameter search, but so far as I can tell (I'm not fluent in sklearn) that seems to be done appropriately.