1
$\begingroup$

I'm new to stats, and I'm researching the pitfalls to avoid when performing Cross Validation (with classification).

Suppose I'm training a classifier, and I have a dataset of 1000 samples, with 1 million features each. I cannot process them all, so I need less features (say, I can compute 300 features). I also have a held-out test set of 100 samples to accurately estimate my out-of-sample, real-world accuracy.

According to this article and this video (found via Data School series), if I filter my 1 million features down to 300, by selecting those features with a highest correlation to the targets of my whole dataset, I am making a mistake (because I'm introducing overfitting which cannot be detected by Cross Validation later on). My held-out set will show this by spitting back a bad accuracy value.

According to the above links, the correct way to do it is to divide my dataset into a training set and Cross-Validation set, and then tune my model (filtering out features, etc) based on this training set and it's associated CV score. If I'm using K-folds, I must tune from scratch each time I make a split/fold, and then average the results.

I understand the reason for all the above, but I would like some clarification on the following:

  • Does it only apply to filtering of features, or does it apply to any feature engineering/model tuning, even those which do not use the entire dataset?

  • When I start using Machine Learning on a new problem and dataset, I usually do the following:

    1. Start by separating out a held-out test set to be used at the very end.
    2. Checking my K-folds accuracy without any feature engineering.
    3. Tune my model and its features.
    4. Check my K-folds accuracy.
    5. Repeat 2-3 until I am satisfied.
    6. Get accuracy on my held-out set.

    Is the above procedure correct, or will I introduce errors and overfitting?

$\endgroup$
2
  • 1
    $\begingroup$ I would have thought that the initial step on a new problem and dataset is to separate out your final testing set, which you will use once at the very end of the process $\endgroup$
    – Henry
    May 7, 2016 at 10:27
  • 1
    $\begingroup$ Oops, you're right, I forgot that step, let me just add it. $\endgroup$ May 7, 2016 at 10:42

1 Answer 1

2
$\begingroup$

 Does it only apply to filtering of features, or does it apply to any feature engineering/model tuning, even those which do not use the entire dataset?

Testing performance can become overoptimistic if test cases somehow enter any kind of calculation during the model training. This can include pre-processing steps (such as your example of filtering by correlation). This applies also if only some test cases leak into the training set (e.g. because some cases have repeated measurements which randomly ended up some in training, some in the test set).

A good rule of thumb is that any kind of calculation that involves more than one case is training and needs to be done on the training set only, so the parameters calculated from the training set can then be applied to the test cases.

Whether you spit your data once (hold out) or repeatedly (cross validation or out-of-bootstrap) is not relevant: Hold out can be subject to bias caused by incorrect splitting and data-driven modelling as well (but it may be easier to avoid programming mistakes with hold out, and steps that need manual control are tedious with repeated splits).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.