1
$\begingroup$

I'm trying to classify 8 types of hand gestures with EMG signals. For that I followed these steps:

  1. Split the entire data into training data and test data
  2. For training data I extracted features. Here how I did that: The training dataset contains 8 files. Each file consists of 50 readings of the Myo armband for a defined gesture. Each reading includes 100 samples per sensor. There are 8 sensors. For every 100 samples/sensor, Mean (of) Absolute Values (MAV) is computed. For 8 sensors, 8 MAVS are computed for a reading. So every row contains 8 MAV values for a particular gesture. Here's a subset of training data (last column is the respective gesture number):

enter image description here

enter image description here

  1. After that I tried to test several ensemble models as classifiers on the training data. For an instance, I tried to use stack of Random Forest, KNN, SVM on the training data. For that I used GridSearchCV for hyper-parameter tuning(I didn't use pipelie). Here's the code:

    param_grid = [

        {
            #Random forest
            'bootstrap': [True, False],
            'max_depth': [40, 50, 60, 70, 80],
            #'max_features': [2, 3],
            'min_samples_leaf': [3, 4, 5],
            'min_samples_split': [8, 10, 12],
            'n_estimators': [10, 15, 20, 25],
            'criterion' : ['gini', 'entropy'],
            'random_state' : [45]
        },
    
        {   
            #K Nearest Neighbours
            'n_neighbors':[5,6,7,9,11],
            'leaf_size':[1,3,5,7],
            'algorithm':['auto', 'ball_tree', 'kd_tree', 'brute'],
            'metric':['euclidean', 'manhattan'] 
    
        },
    
        {
            #SVM
            'C': list(np.arange(1, 5, 0.01)),
            'gamma': ['scale', 'auto'],
            'kernel': ['rbf', 'poly', 'sigmoid', 'linear'],
            'decision_function_shape': ['ovo', 'ovr'],
            'random_state' : [45]
        }    
    ] 
    
    models_to_train = [RandomForestClassifier(), KNeighborsClassifier(), svm.SVC()]
    
    final_models = []
    for i, model in enumerate(models_to_train):
        params = param_grid[i]
    
        clf = GridSearchCV(estimator=model, param_grid=params, cv=20, scoring = 'accuracy').fit(data_train, label_train)
        final_models.append(clf.best_estimator_)
    
  2. Did similar feature extraction procedure like step 2 for test data

  3. Fit the stacked model to the training data, made prediction on test data and calculated accuracy.

estimators = [
    ('rf', final_models[0]),
    ('knn', final_models[1])                 
]
clf = StackingClassifier(
    estimators=estimators, final_estimator=final_models[2]
)

category_predicted = clf.fit(data_train, label_train).predict(data_test)

acc = accuracy_score(label_test, category_predicted) * 100

Now, my question is,

Is there any chance of data leakage in this procedure?

Edit

I believe this procedure suffers from data leakage because I did feature extraction in step 2 on entire training data and those features are used in GridSearchCV without any pipeline . If I put feature extraction (what described in step 2) and estimator in pipeline (as discussed here: https://towardsdatascience.com/pre-process-data-with-pipeline-to-prevent-data-leakage-during-cross-validation-e3442cca7fdc), then it can be avoided.

$\endgroup$
2
  • $\begingroup$ I'm not completely sure what you mean by "MAV" in your Step 2, where you say that you did feature extraction. Please provide more details on that. $\endgroup$
    – EdM
    Aug 24 '20 at 22:32
  • $\begingroup$ I added definition of MAV (Mean (of) Absolute Values) in question. $\endgroup$
    – Debbie
    Aug 24 '20 at 22:53
1
$\begingroup$

A potentially bigger problem than data leakage between test and training sets is the unreliability of test/train splits of small data sets. You need many thousands of cases for that to be reliable. Otherwise you are throwing away information by limiting the size of the training set, and you are getting imprecise estimates of model validity by having too small a test set.

As you have a well-defined algorithm for building your model, consider internal validation by bootstrapping instead. This answer outlines the procedure. Even if you continue to use a train/test split for your modeling, repeating the entire modeling process including the original train/test split on multiple bootstrap samples of your data will evaluate how much of a problem is imposed by any data leakage.

In your case you don't appear to have a problem with data leak in your cross validation. All you have done is to combine raw readings into a type of average, the MAV, without any attempt to standardize the readings within each sensor at that point of the analysis. There might be some standardization later on within your parameter search, but so far as I can tell (I'm not fluent in sklearn) that seems to be done appropriately.

That's different from the situation described in the page you link. There each of the predictors was standardized from the beginning to put them on the same relative scale, as is needed for principal-component analyses and penalized regressions (ridge, LASSO). As the degree of transformation of any predictor needed for the standardization will vary from sample to sample, that can be a problem if your (wise) intent is to repeat the entire modeling process (including the standardization) in each CV fold or bootstrap sample. You haven't fed your parameter search pre-standardized data, just an average without any change of scale, so you shouldn't have that problem.

$\endgroup$
2
  • $\begingroup$ I found an implementation of Bootstrap CV here : kaggle.com/sasha18/resampling-methods-using-bootstrap-cv But is it possible (or practically feasible) to tune huge number of hyper-parameters with Bootstrap CV which I did easily with GridSearchCV? $\endgroup$
    – Debbie
    Aug 27 '20 at 11:26
  • 1
    $\begingroup$ @Debbie I'd recommend a different approach. Use GridSearchCV on the whole data set to choose hyper-parameters and build the full model: no test/train split. Then set up your bootstrapped re-samples, and do the GridSearchCV and model building on each of the re-samples. On each of the re-samples you have the advantages you found with GridSearchCV. Then test each of the resulting models on the full data set. This estimates the quality of the modeling process, including the GridSearchCV, without a fixed test/train split. See this page. $\endgroup$
    – EdM
    Aug 27 '20 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.