I'm new to stats, and I'm researching the pitfalls to avoid when performing Cross Validation (with classification).
Suppose I'm training a classifier, and I have a dataset of 1000 samples, with 1 million features each. I cannot process them all, so I need less features (say, I can compute 300 features). I also have a held-out test set of 100 samples to accurately estimate my out-of-sample, real-world accuracy.
According to this article and this video (found via Data School series), if I filter my 1 million features down to 300, by selecting those features with a highest correlation to the targets of my whole dataset, I am making a mistake (because I'm introducing overfitting which cannot be detected by Cross Validation later on). My held-out set will show this by spitting back a bad accuracy value.
According to the above links, the correct way to do it is to divide my dataset into a training set and Cross-Validation set, and then tune my model (filtering out features, etc) based on this training set and it's associated CV score. If I'm using K-folds, I must tune from scratch each time I make a split/fold, and then average the results.
I understand the reason for all the above, but I would like some clarification on the following:
Does it only apply to filtering of features, or does it apply to any feature engineering/model tuning, even those which do not use the entire dataset?
When I start using Machine Learning on a new problem and dataset, I usually do the following:
- Start by separating out a held-out test set to be used at the very end.
- Checking my K-folds accuracy without any feature engineering.
- Tune my model and its features.
- Check my K-folds accuracy.
- Repeat 2-3 until I am satisfied.
- Get accuracy on my held-out set.
Is the above procedure correct, or will I introduce errors and overfitting?