In machine learning, we perform feature engineering and selections in pipelines and crossvalidate to obtain results in order to avoid data leakage and avoid introducing prior knowledge into the learner.

Now, in a project I’m working on, I am starting to do some correlation analysis to create a subset of the features to work on in terms of feature engineering.

If I just take the whole dataset and calculate the correlation of each feature with the target feature, haven’t i technically introduced some prior knowledge into my process? Should I just calculate correlations between the features using the training set? If yes, what is technically the purpose of doing so since I’m not really evaluating the outcome of correlation analysis on a test set?


Yes, you introduce some bias when you pick features based on correlation with the target feature.

However, it's not that bad because

  1. After you pick your features, you should perform normalization to put them on an equal footing.

  2. In order to successfully learn the target function, you have to pick good features. The bias introduced by choosing features based on correlation is usually a necessary sacrifice to pick good features.

Yes, you should only calculate correlations between the features using the training set. If you were to calculate correlations using the test set as well, your testing error would have an optimistic bias. You want the test set error to be unbiased so you can get a good estimate of how well your model approximates the target function.

  • $\begingroup$ Thank you. Just for full disclosure, I have already made a full model evaluation and comparison run with nested CV and preprocessing features. I have selected two promising models and based on your input I will use the same training set to look for correlations. $\endgroup$ – Odisseo Apr 1 at 0:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.