1
$\begingroup$

I have 117 samples which I used to select and train a model.

What I did: 1) pre-processed the 117 samples (normalization, statistics, etc); 2) created 4 folds (random split); 3) performed a nested-repated CV in six algorithms; 4) selected the best

With this I saw that my model had good performance and decided to take more samples.

Now I have an extra 30 samples and I'm unsure what to do.

What I tried: I merged all the data (117 + 30) and did the pre-processing. Then, I separated in two datasets: the training (117 samples) and the testing (the 30 extra samples). I trained the best model again with the best parameters using the training set and tested it in the testing set.

Is this the right thing to do?

$\endgroup$

1 Answer 1

0
$\begingroup$

In the preprocessing step, you shouldn't mix train/validation or train/test data. It's data leakage, and it may (or may not) have serious effects depending on the data. So, for example, say you're doing mean/std standardisation. You'll compute the mean/std from your training set (117 samples), and apply those estimated mean/std for standardising your test set. Same methodology should be applied in cross validation steps.

$\endgroup$
2
  • $\begingroup$ Even for normalization? Since this step only put the values in a range of 0 to 1, I didn't thought it was considered data leakage $\endgroup$ Commented Mar 21, 2021 at 0:22
  • 1
    $\begingroup$ Yes, in principle, even for that. Because in actual testing, you wouldn't know the max value you'll get in the test set. $\endgroup$
    – gunes
    Commented Mar 21, 2021 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.