0
$\begingroup$

I want to generate some summary statistics and look at the correlation between the variables of my dataset to remove certain features (very low variance, very high correlation).

The dataset is the famous Titanic dataset from Kaggle, where I have a CSV for training & testing with both X and y and a second "prediction" CSV for which I only have X.

My question is, can I concatenate the two datasets (train/test + prediction) to generate the summary statistics and calculate the correlations between the variables of X or should I only use the training set?

Improve this question
$\endgroup$
1
  • $\begingroup$ What is "low variance"? Multiply your data by 999999 and each variable would have high variance... Using summary statistics to remove features is a bad idea. $\endgroup$
    – Tim
    Commented Nov 22, 2018 at 19:34

1 Answer 1

Reset to default
0
$\begingroup$

From my experience, you should really not consolidate the test and the training set. There is a reason that we keep them separated and that is to construct a robust and sound model.

As a general rule, with real world data, whatever data preprocessing decisions you make should be based only on the training set. If you feel that you do not have enough data to create two sets then there are other solutions (bootstrapping, CV) that can help you with that.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If I use CV, I still have a "prediction" dataset, can I use the information from that dataset for preprocessing decisions? $\endgroup$
    – Robert Patchouli
    Commented Nov 22, 2018 at 17:22
  • $\begingroup$ When using CV the idea is to randomise on each iteration the training set and the valuation set, while keeping their sizes the same 60/30 (and 10 for testing). Or whatever other ratio you prefer. I wouldn't touch the prediction dataset. $\endgroup$
    – Jespar
    Commented Nov 22, 2018 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.