1
$\begingroup$

Let's say I want to test the performance of my dimension reduction + classification pipeline. To do this, I will use k-fold cross validation. I know that performing dimension reduction on the complete dataset before creating the k folds is bad due to overfitting. To avoid this, the k folds for training and testing are created first. My question is the following: how should my dimension reduction + classification pipeline learn? I see two options:

  1. Take my training data, divide it in two (how many samples go to each is to be determined). Use one subset to learn the dimension reduction mapping. Then, pass the other set through the learned mapping and use the reduced features to learn the classifier. Now, none of the steps have overfitted.
  2. Take my training data, apply my dimension reduction to it i.e use the same data for learning and reducing. Use the reduced data to learn the classifier.

I tend to prefer approach (1) given that no overfitting occurs. With method (2), I would run into issues when I want to use my dimension reduction + classifier pipeline on new data.

Is approach (1) the correct one? Is there another way to do this? I'm not making assumptions on whether the dimension reduction is supervised or not.

$\endgroup$
1
$\begingroup$

If you use approach 1, you'll need to reserve a third independent set for validation/testing of the ready-to-use model. Other than that, it is perfectly finde.

People usually try to avoid that additional split (you'd need yet another one if you do data-driven hyperparameter optimiziation!) and therefore go for approach 2 which leaves the second split for validation/testing.

$\endgroup$
  • $\begingroup$ Yeah , that makes sense, but wouldn't the second approach be overfitted? If the dimension reduction finds features correlated with labels, then learning and reducing with the same data will be a problem for the classifier as it won't generalize. Regarding validation/testing, I mention at the beginning that I already divide the data in kfolds for training and testing $\endgroup$ – Damian Jun 14 '16 at 11:49
  • $\begingroup$ Well, the trade-off is your judgement of what is worse: having an overfit model because feature reduction and parameter estimation were done on the same (but larger) data, or having an overfit model because the data sets for feature reduction and model parameter estimation were smaller (e.g. half the sample size compared to the other approach). $\endgroup$ – cbeleites supports Monica Jun 15 '16 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.